The bigger concern is how large the git history is going to get on the repository.
btown 1 hours ago [-]
I recall that this became a big problem for the Homebrew project in terms of load on the repo, to the extent that Github asked them not to recommend/default-enable shallow clones for their users: https://github.com/Homebrew/brew/issues/15497#issuecomment-1...
This is likely to be lower traffic, and the history should (?) scale only linearly with new data, so likely not the worst thing. But it's something to be cognizant of when using SCM software in unexpected ways!
roncesvalles 1 hours ago [-]
How would shallow clone be more stressful for GitHub than a regular clone?
enchilada 58 minutes ago [-]
Shallow clones (and the resulting lack of shared history data) break many assumptions that packfile optimisations rely on.
This makes more sense. I still wonder if the author isn't just effectively recreating Apache Iceberg manually here.
tomrod 2 hours ago [-]
Are they paying for the repo space, I wonder?
cyanydeez 1 hours ago [-]
someones paying to keep name dropping Iceberg(tm)
zerocrates 2 hours ago [-]
"The dataset is organized as one Parquet file per calendar month, plus 5-minute live files for today's activity. Every 5 minutes, new items are fetched from the source and committed directly as a single Parquet block. At midnight UTC, the entire current month is refetched from the source as a single authoritative Parquet file, and today's individual 5-minute blocks are removed from the today/ directory."
So it's not really one big file getting replaced all the time. Though a less extreme variation of that is happening day to day.
tomrod 2 hours ago [-]
Parquet is a very efficient storage approach. Data interfaces tend to treat paths as partitions, if logical.
fabmilo 2 hours ago [-]
Was thinking the same thing. probably once a day would be more than enough.
if you really want a minute by minute probably a delta file from the previous day should be more than enough.
brtkwr 2 hours ago [-]
This comment should make it into the download in a few mins.
tantalor 2 hours ago [-]
As should this reply
ericfr11 1 hours ago [-]
Hello to myself for prosperity
christophilus 36 minutes ago [-]
And also for posterity.
nostrapollo 10 minutes ago [-]
I'll live on, posthumously
culi 11 minutes ago [-]
' OR 1=1 --
culi 11 minutes ago [-]
<script>alert('xss')</script>
epogrebnyak 2 hours ago [-]
Wonder why median votes count is 0, seems every post is getting at least a few votes - maybe this was not the case in the past
epogrebnyak 2 hours ago [-]
Ahhh I get it the moment I asked, there are usually no votes on comments
mlhpdx 3 hours ago [-]
Static web content and dynamic data?
> The archive currently spans from 2006-10 to 2026-03-16 23:55 UTC, with 47,358,772 items committed.
That’s more than 5 minutes ago by a day or two. No big deal, but a little bit depressing this is still how we do things in 2026.
voxic11 2 hours ago [-]
That is just the archive part, if you just would finish reading the paragraph you would know that updates since 2026-03-16 23:55 UTC are "are fetched every 5 minutes and committed directly as individual Parquet files through an automated live pipeline, so the dataset stays current with the site itself."
So to get all the data you need to grab the archive and all the 5 minute update files.
not really since original comment completely missed it
john_strinlai 51 minutes ago [-]
not to be "that guy" but it is pretty explicitly laid out in the guidelines, with an example and everything
xandrius 2 hours ago [-]
I don't get what you meant with this comment.
john_strinlai 2 hours ago [-]
the data updates every 5 minutes, but the description on huggingface says the last update was 2 days ago.
they are suggesting that the huggingface description should be automatically updating the date & item count when the data gets updated.
voxic11 2 hours ago [-]
No that is the date at which the bulk archive ends and the 5 minute update files begin, so it should not be updated.
2 hours ago [-]
maxloh 2 hours ago [-]
Could you also release the source code behind the automatic update system?
imhoguy 2 hours ago [-]
Yay! So much knowledge in just 11GB. Adding to my end of the World hoarding stash!
sockaddr 14 minutes ago [-]
Your family is starving and your dog died of radiation poisoning from the fallout but at least your local LLM can browse this and recommend a good software stack for your automated booby traps.
gkbrk 4 hours ago [-]
My Hacker News items table in ClickHouse has 47,428,860 items, and it's 5.82 GB compressed and 18.18 GB uncompressed. What makes Parquet compression worse here, when both formats are columnar?
0cf8612b2e1e 4 hours ago [-]
Sorting, compression algorithm +level, and data types can all have an impact. I noted elsewhere that a Boolean is getting represented as an integer. That’s one bit vs 1-4 bytes.
There is also flexibility in what you define as the dataset. Skinnier, but more focused tables could be space saving vs a wide table that covers everything -will probably break compressible runs of data.
xnx 4 hours ago [-]
Parquet has a few compression option. Not sure which one they are using.
hirako2000 4 hours ago [-]
Plus isn't the least wasteful format, native duckdb for instance compacts better. That's not just down to the compression algorithm, which as you say got three main options for parquet.
boznz 22 minutes ago [-]
.. and Remove all the political shit-slop since COVID/AI and it's probably under a gig.
0cf8612b2e1e 4 hours ago [-]
Under the Known Limitations section
deleted and dead are integers. They are stored as 0/1 rather than booleans.
Is there a technical reason to do this? You have the type right there.
kshacker 3 hours ago [-]
Good for demo but every 5 minutes? Why?
Imustaskforhelp 2 hours ago [-]
It can have some good use cases I can think of. Personally I really appreciate the 5 minute update.
alstonite 3 hours ago [-]
What happened between 2023 and 2024 to cause the usage dropoff?
ghgr 3 hours ago [-]
I'd say it's less a usage dropoff and more a reversion to the mean after Covid
tehjoker 3 hours ago [-]
That's a possible hypothesis, but there was also a rising trend prior, it wasn't stable.
imhoguy 2 hours ago [-]
Return to office
robotswantdata 2 hours ago [-]
Where’s the opt out ?
BowBun 1 hours ago [-]
By posting comments on this site, you are relinquishing your right to that content. It belongs to YC and it is theirs to enforce, not yours. https://www.ycombinator.com/legal/
robotswantdata 1 hours ago [-]
Max Schrems would like a word
john_strinlai 2 hours ago [-]
hackernews is very upfront that they do not really care about deletion requests or anything of that sort, so, the opt out is to not use hackernews.
ratg13 2 hours ago [-]
Create a new account every so often, don’t leave any identifying information, occasionally switch up the way you spell words (British/US English), and alternate using different slang words and shorthand.
GeoAtreides 53 minutes ago [-]
funnily enough, if everyone did this (at least make a new account often), it would prove more destructive to what HN (purposefully) wants to do than deleting the occasional account data
fdghrtbrt 2 hours ago [-]
And do what I do - paste everything into ChatGPT and have it rephrase it. Not because I need help writing, but because I’d rather not have my writing style used against me.
socksy 2 hours ago [-]
I can't stand this and will actively discriminate against comments I notice in that voice. Even this one has "Not because [..], but because [..]"
culi 8 minutes ago [-]
Look up "adversarial stylometry"
coppsilgold 1 hours ago [-]
This just gives OpenAI that data.
Perhaps you could use a local translation model to rephrase (such as TranslateGemma). If translating English to English doesn't achieve this effect then use an intermediate language, one the model is good at to not mangle meaning too much.
fdghrtbrt 1 hours ago [-]
I run Qwen 3 locally, but I mention OpenAI on HN so people understand what I’m referring to.
I'm actually working on a browser extension to do just this with adversarial stylometry techniques
tantalor 2 hours ago [-]
The back button
palmotea 4 hours ago [-]
> At midnight UTC, the entire current month is refetched from the source as a single authoritative Parquet file, and today's individual 5-minute blocks are removed from the today/ directory.
Wouldn't that lose deleted/moderated comments?
BoredPositron 3 hours ago [-]
I guess that's the point.
Imustaskforhelp 2 hours ago [-]
Can't someone create an automatic script which can just copy the files say 5 minutes before midnight UTC?
This is great. I've soured on this site over the past few years due to the heavy partisanship that wasn't as present in the early days (eternal September), but there are still quite a few people whose opinions remain thought-provoking and insightful. I'm going to use this corpus to make a local self-hosted version of HN with the ability to a) show inline article summaries and b) follow those folks.
Imustaskforhelp 2 hours ago [-]
As someone who had made a project analysing hackernews who had used clickhouse, I really feel like this is a project made for me (especially the updated every 5 minute aspect which could've helped my project back then too!)
Your project actually helps me out a ton in making one of the new project ideas that I had about hackernews that I had put into the back-burner.
I had thought of making a ping website where people can just @Username and a service which can detect it and then send mail to said username if the username has signed up to the service (similar to a service run by someone from HN community which mails you everytime someone responds to your thread directly, but this time in a sort of ping)
[The previous idea came as I tried to ping someone to show them something relevant and thought that wait a minute, something like ping which mails might be interesting and then tried to see if I can use algolia or any service to hook things up but not many/any service made much sense back then sadly so I had the idea in back of my mind but this service sort of solves it by having it being updated every 5 minutes]
Your 5 minute updates really make it possible. I will look what I can do with that in some days but I am seeing some discrepancy in the 5 minute update as last seems to be 16 march in the readme so I would love to know more about if its being updated every 5 minutes because it truly feels phenomenal if true and its exciting to think of some new possibilities unlocked with it.
tonymet 3 hours ago [-]
what's the license for HN content?
BowBun 1 hours ago [-]
We have LLMs and links to TOS, this is easily answerable by _anyone_ on the internet at this point.
Comments+posts are defined as user generated content, you have no right to its privacy/control in any capacity once you post it - https://www.ycombinator.com/legal/
YC in theory has the right to go after unauthorized 3rd parties scraping this data. YC funds startups and is deeply vested in the AI space. Why on Earth would they do that.
echelon 3 hours ago [-]
At this point, you can train on anything without repercussion.
Copyright doesn't seem to matter unless you're an IP cartel or mega cap.
marginalia_nu 3 hours ago [-]
Laughs nervously in jurisdiction without fair use doctrine
Onavo 4 hours ago [-]
Is is possible to only download a subset? e.g. Show HNs or HN Whoishiring. The Show HNs and HN Whoishiring are very useful for classroom data science i.e. a very useful set of data for students to learn the basic of data cleaning and engineering.
nelsondev 4 hours ago [-]
It’s date partitioned, you could download just a date range. It’s also parquet, so you can download just specific columns with the right client
bstsb 4 hours ago [-]
what’s the license? “do whatever the fuck you want with the data as long as you don’t get caught”? or does that only work for massive corporations
BoredPositron 3 hours ago [-]
The universal license.
GeoAtreides 4 hours ago [-]
is the legal page a placeholder, do words have no meaning?
Mods, enforce your license terms, you're playing fast and loose with the law (GDPR/CPRA)
Retr0id 4 hours ago [-]
Which terms are not being enforced? (not disagreeing I just don't feel like reading a large legal document)
GeoAtreides 4 hours ago [-]
> By uploading any User Content you hereby grant and will grant Y Combinator and its affiliated companies
The user content is supposed to be licensed only Y Combinator and (bleah) its affiliated companies (which are many, all the startups they fund, for example).
jmalicki 3 hours ago [-]
Curious why it should be on HackerNews to enforce restrictions on content they only license from you?
If it's owned by you and only licensed by HN shouldn't you be the one enforcing it?
AndrewKemendo 3 hours ago [-]
Seems like they are trying to do that through the stated legal intermediary (YC)
zamadatix 3 hours ago [-]
If you carry on the quote two more words:
> ... a nonexclusive
I.e. this section is talking to additional rights to the content you post to ALSO go to YC, not that YC is guaranteeing it (+friends) will be the only one to hold these rights or will enforce who else should hold the rights to your publicly shared content for you.
There's a more intricate conversation to be had with GDPR and public data on forums in general but that's wholly unrelated to what YC's legal page says and still unlikely to end up in an alarming result.
Bewelge 1 hours ago [-]
I think that's incorrect. Exclusivity would be something you grant to YC. These terms need to make sense to be valid. Claiming exclusive rights would mean they are forbidding YOU from licensing YOUR rights to anyone else.
Imagine Facebook claiming that by uploading images you are granting them exclusive usage rights to that image. It would mean you couldn't upload it to any other site with similar terms anymore.
ryandvm 3 hours ago [-]
That agreement is largely about "Personal Information", not the posts and comments.
That said, there are "no scraping" and "commercial use restricted" carve-outs for the content on HN. Which honestly is bullshit.
ungruntled 4 hours ago [-]
None that I could see:
Your submissions to, and comments you make on, the Hacker News site are not Personal Information and are not "HN Information" as defined in this Privacy Policy.
Other Users: certain actions you take may be visible to other users of the Services.
GeoAtreides 4 hours ago [-]
I mean, just because they say the comments are not PI doesn't make it so.
ungruntled 4 hours ago [-]
That’s a good point. I’m only referring to the terms they used in the privacy policy.
ryandvm 3 hours ago [-]
Eh, fuck that agreement. I'm kind of old school in that I believe if you put it on the internet without an auth-wall, people should be allowed to do whatever they want with it. The AI companies seem to agree.
Then again, I'm not the guy that is going to get sued...
hrmtst93837 52 minutes ago [-]
Legal theory about public data is fun right up until someone with money decides their ToS mean something and files suit, because courts are usually a lot less impressed by "I could access it in my browser" once you pulled millions of records with a scraper. Scrape if you want, just assume you're buying legal risk.
Ylpertnodi 3 hours ago [-]
> I believe if you put it on the internet without an auth-wall, people should be allowed to do whatever they want with it.
I agree.
It's the owners of the sites that have to follow rules, not us.
hrmtst93837 2 hours ago [-]
[dead]
kmeisthax 3 hours ago [-]
"I'm kind of old school in that I believe if you put grass on the ground without a fence, people should be allowed to do whatever they want with it. The noblemen with a thousand cows seem to agree."
And that, my friends, is how you kill the commons - by ignoring the social context surrounding its maintenance and insisting upon the most punitive ways of avoiding abuse.
petercooper 3 hours ago [-]
Context is important, but isn’t HN’s social context, in particular, that the site is entirely public, easily crawled through its API (which apparently has next to no rate limits) and/or Algolial, and has been archived and mirrored in numerous places for years already?
echelon 3 hours ago [-]
Signal and information are not grass.
Grass and property require upkeep. Radio waves and electromagnetic radiation do not.
I don't want your dog to piss on my lawn and kill my grass. But what harm does it cause me if you take a picture of my lawn? Or if I take a picture of your dog?
If I spend $100M making a Hollywood movie - pay employees, vendors, taxes - contribute to the economic growth of the country - and then that product gets stolen and given away completely for free without being able to see upside, that's a little bit different.
But my Hacker News comment? It's not money.
I think there are plausible ways to draw lines that protect genuine work, effort, and economics while allowing society and innovation to benefit from the commons.
hsuduebc2 4 hours ago [-]
How is is he breaking gdpr here?
andrewmcwatters 4 hours ago [-]
They already refuse to comply with CPRA, instead electing to replace your username with a random 6(?) character string, prefixed with `_`, if I remember correctly.
I know, because I've been here since maybe 2015 or so, but this account was created in 2019.
So any PII you have mentioned in your comments is permanent on Hacker News.
I would appreciate it if they gave users the ability to remove all of their personal data, but in correspondence and in writing here on Hacker News itself, Dan has suggested that they value the posterity of conversations over the law.
stopbulying 3 hours ago [-]
[dead]
lokimoon 3 hours ago [-]
You are the product
waynesonfire 2 hours ago [-]
Your reward is the endorphin hit from writing this comment.
trwhite 46 minutes ago [-]
Hello. I didn’t consent to any of my HN comments being used in this way. Please kindly remove them.
I’m reading that paragraph now and fail to see anything about a relationship with huggingface or the user responsible for copying the data.
cj 36 minutes ago [-]
To be incredibly pedantic to the point of being irrelevant: technically the sign up page 1) doesn't have a clickwrap "I agree" checkbox, and 2) there's no link to the TOS on the sign up page.
That makes the implicit TOS agreement legally confusing depending on jurisdiction.
(Not that it really matters, but I find these technicalities amusing)
s0ss 37 minutes ago [-]
Only Y Combinator and its affiliated companies have license, me thinks.
trwhite 34 minutes ago [-]
@dang What’s Hacker News’ official stance on this?
The bigger concern is how large the git history is going to get on the repository.
This is likely to be lower traffic, and the history should (?) scale only linearly with new data, so likely not the worst thing. But it's something to be cognizant of when using SCM software in unexpected ways!
See also: https://github.com/orgs/Homebrew/discussions/225
So it's not really one big file getting replaced all the time. Though a less extreme variation of that is happening day to day.
> The archive currently spans from 2006-10 to 2026-03-16 23:55 UTC, with 47,358,772 items committed.
That’s more than 5 minutes ago by a day or two. No big deal, but a little bit depressing this is still how we do things in 2026.
So to get all the data you need to grab the archive and all the 5 minute update files.
archive data is here https://huggingface.co/datasets/open-index/hacker-news/tree/...
update files are here (I know that its called "today" but it actually includes all the update files which span multiple days at this point) https://huggingface.co/datasets/open-index/hacker-news/tree/...
probably uncalled for
they are suggesting that the huggingface description should be automatically updating the date & item count when the data gets updated.
There is also flexibility in what you define as the dataset. Skinnier, but more focused tables could be space saving vs a wide table that covers everything -will probably break compressible runs of data.
Perhaps you could use a local translation model to rephrase (such as TranslateGemma). If translating English to English doesn't achieve this effect then use an intermediate language, one the model is good at to not mangle meaning too much.
sample content from users on this page: https://news.ycombinator.com/leaders
and ask the LLM to rephrase it in their voice
Wouldn't that lose deleted/moderated comments?
Your project actually helps me out a ton in making one of the new project ideas that I had about hackernews that I had put into the back-burner.
I had thought of making a ping website where people can just @Username and a service which can detect it and then send mail to said username if the username has signed up to the service (similar to a service run by someone from HN community which mails you everytime someone responds to your thread directly, but this time in a sort of ping)
[The previous idea came as I tried to ping someone to show them something relevant and thought that wait a minute, something like ping which mails might be interesting and then tried to see if I can use algolia or any service to hook things up but not many/any service made much sense back then sadly so I had the idea in back of my mind but this service sort of solves it by having it being updated every 5 minutes]
Your 5 minute updates really make it possible. I will look what I can do with that in some days but I am seeing some discrepancy in the 5 minute update as last seems to be 16 march in the readme so I would love to know more about if its being updated every 5 minutes because it truly feels phenomenal if true and its exciting to think of some new possibilities unlocked with it.
Comments+posts are defined as user generated content, you have no right to its privacy/control in any capacity once you post it - https://www.ycombinator.com/legal/
YC in theory has the right to go after unauthorized 3rd parties scraping this data. YC funds startups and is deeply vested in the AI space. Why on Earth would they do that.
Copyright doesn't seem to matter unless you're an IP cartel or mega cap.
https://www.ycombinator.com/legal/
Mods, enforce your license terms, you're playing fast and loose with the law (GDPR/CPRA)
The user content is supposed to be licensed only Y Combinator and (bleah) its affiliated companies (which are many, all the startups they fund, for example).
If it's owned by you and only licensed by HN shouldn't you be the one enforcing it?
> ... a nonexclusive
I.e. this section is talking to additional rights to the content you post to ALSO go to YC, not that YC is guaranteeing it (+friends) will be the only one to hold these rights or will enforce who else should hold the rights to your publicly shared content for you.
There's a more intricate conversation to be had with GDPR and public data on forums in general but that's wholly unrelated to what YC's legal page says and still unlikely to end up in an alarming result.
Imagine Facebook claiming that by uploading images you are granting them exclusive usage rights to that image. It would mean you couldn't upload it to any other site with similar terms anymore.
That said, there are "no scraping" and "commercial use restricted" carve-outs for the content on HN. Which honestly is bullshit.
Your submissions to, and comments you make on, the Hacker News site are not Personal Information and are not "HN Information" as defined in this Privacy Policy.
Other Users: certain actions you take may be visible to other users of the Services.
Then again, I'm not the guy that is going to get sued...
I agree. It's the owners of the sites that have to follow rules, not us.
And that, my friends, is how you kill the commons - by ignoring the social context surrounding its maintenance and insisting upon the most punitive ways of avoiding abuse.
Grass and property require upkeep. Radio waves and electromagnetic radiation do not.
I don't want your dog to piss on my lawn and kill my grass. But what harm does it cause me if you take a picture of my lawn? Or if I take a picture of your dog?
If I spend $100M making a Hollywood movie - pay employees, vendors, taxes - contribute to the economic growth of the country - and then that product gets stolen and given away completely for free without being able to see upside, that's a little bit different.
But my Hacker News comment? It's not money.
I think there are plausible ways to draw lines that protect genuine work, effort, and economics while allowing society and innovation to benefit from the commons.
I know, because I've been here since maybe 2015 or so, but this account was created in 2019.
So any PII you have mentioned in your comments is permanent on Hacker News.
I would appreciate it if they gave users the ability to remove all of their personal data, but in correspondence and in writing here on Hacker News itself, Dan has suggested that they value the posterity of conversations over the law.
https://www.ycombinator.com/legal/
See: User Content Transmitted Through the Site
That makes the implicit TOS agreement legally confusing depending on jurisdiction.
(Not that it really matters, but I find these technicalities amusing)