UPDATE (February 4, 2024): This is the discussion about this project on HN: here. Please specifically read @dang's comment regarding the core assumption of this project: here. On a personal note, the number of Stories removed yesterday (Saturday, February 3, 2024) was the lowest ever recorded by the service. This includes 2 duplicate Stories. As a side note, in the list always check whether a Story is a duplicate or not: this is a very reasonable reason for removal and unfortunately I have no way of automatically determining it in the service!
The purpose of this project is to try to understand the type and scale of the moderation of the Hacker News Front Page.
NOTE: I love Hacker News. I try to read it every day. In the case of OnnxStream (here for example), 95% of the comments were helpful and intelligent. I also understand that moderating a site with huge traffic and where users are basically anonymous must be a very difficult task.
Returning to the purpose of this project, from what I have been able to see, the "public" (i.e. observable from the outside) moderation of the Front Page consists of two main tools: modification of the title of a Story (voluntarily or involuntarily influencing its growth in terms of rank) or directly its removal.
Regarding the first type of moderation, an excellent site is already available that tracks changes to Story titles. Here instead I will focus on the second type.
For the reasons explained in the "Why?" section below, I have developed a small application that logs all the Stories that are removed from the Front Page, for personal use. I later discovered that there is no tool/website that provides this type of information and I decided to make it public here. It was a difficult decision but my rationale is: is it better to have more transparency or less transparency?
If you know of a tool/website similar to this, please let me know: I will archive this repo or set it to private.
A possible very positive outcome for this project could be to have a list similar to this, but available directly among the HN lists. Or even to notify a user when a Story is penalized on the Front Page, perhaps indicating the number of flags and/or the reason, for example.
Feel free to skip this part or click to expand
A friend of mine posted two Stories on Hacker News related to OnnxStream (31 days apart), the first related to SDXL Turbo support and the second related to TinyLlama and Mistral 7B support.
In the case of the first, the Story was among the first on the Front Page, until its title was changed from "Stable Diffusion Turbo on a Raspberry Pi Zero 2 generates an image in 29 minutes" to "OnnxStream: Stable Diffusion XL 1.0 Base on a Raspberry Pi Zero 2". This effectively "killed" the Story. One user pointed out that the new title didn't reflect the spirit of the Story (thanks @practice9).
In the case of the second, the Story was in third place on the Front Page, less than an hour after the submission. In this case it was simply removed from the Front Page.
Having discovered this, perplexed, I sent an email to the moderator. @dang, who was very kind and quick in his response, explained to me that the Story had been flagged by users even without being explicitly [flagged], and that he could therefore only hypothesize the causes of the flag. His hypothesis was that (some?) users might be fed up with news related to LLMs.
While I have no reason to doubt Daniel's good faith, it's hard to believe that HN users would be tired of LLM-related news.
So I decided to develop a small console application to determine the frequency of this phenomenon (actually I was also motivated by the prospect of writing some C# code, after more than 2 years of complete abstinence). I subsequently discovered that there were no tools/websites that monitored this specific phenomenon and I therefore decided to make it public here.
Using the official HN API, the service fetches 90 Top Stories every minute and makes a comparison with the first 30 Top Stories (i.e. the Front Page) fetched the previous minute. It logs all missing Stories here. The assumption is that a Story cannot go from the top 30 to a position greater than 90 in a single minute, without having been explicitly removed. If a Story reappears on the Front Page, it is removed from this log. All Stories present in the second-chance pool are excluded from the log. Title and URL are those from when the Story first appeared in the top 30. The number of points and comments and the rank are those from when the Story was removed from the Front Page. The ID points to the news.social-protocols.org page for that Story, which provides a graph of the Story's position on the Front Page over time.
NOTE: always check whether a Story is a duplicate or not: this is a very reasonable reason for removal and unfortunately I have no way of automatically determining it in the service!
- 47648854 #30 16 points 2 comments -> Progressive encoding and decoding of 'repeated' protobuffer fields
- 47711964 #27 33 points 20 comments -> Sam Altman's Coworkers Say He Can Barely Code and Misunderstands Basic Concepts
- 47714239 #8 33 points 13 comments -> The tool that won't let AI say anything it can't cite
- 47714634 #5 15 points 2 comments -> How Uv Works Under the Hood
- 47715618 #26 12 points 0 comments -> Untaxed hidden wealth surpasses wealth of the poorest half of humanity
- 47715530 #18 3 points 0 comments -> We're Getting the Wrong Message from Mythos
- 47716219 #27 8 points 4 comments -> Python Is Dead
- 47716412 #1 334 points 94 comments -> Microsoft suspends dev accounts for high-profile open source projects
- 47716043 #7 746 points 359 comments -> France Launches Government Linux Desktop Plan as Windows Exit Begins
- 47718718 #18 13 points 0 comments -> Consumer sentiment plummets to record low as Iran war jacks up inflation
- 47718830 #4 113 points 1 comments -> CPU-Z and HWMonitor Compromised
- 47721022 #9 6 points 0 comments -> How Social Media Feminised Us All
- 47719526 #28 -> Names and faces of those killed by Israel in its April 8 massacre
- 47721874 #14 4 points 2 comments -> Claude AI Assistant for Microsoft Office
- 47721516 #22 13 points 3 comments -> "Not Even Government Agencies" - Proton's misleading marketing
- 47727387 #18 33 points 40 comments -> The Seasons Are Wrong
- 47728134 #21 17 points 6 comments -> Moooooonitoring the Cow.txt Herd
- 47728721 #23 26 points 40 comments -> Artemis II is competency porn
- 47730963 #25 8 points 0 comments -> Brazil seizes over 1,100 weapons and 1.5 tons of drugs from US, says official
- 47733385 #22 9 points 20 comments -> Show HN: Editing 2000 photos made me build a macOS bulk photo editor
- 47734224 #12 7 points 0 comments -> Democratic AI to serve the public – OneProject.org
- 47735101 #5 21 points 15 comments -> AI Job Loss Tracker
- 47735704 #18 61 points 43 comments -> We spoke to the man making viral Lego-style AI videos for Iran
- 47735507 #25 10 points 1 comments -> Used Graphify to turn incidents into a queryable knowledge graph
- 47671365 #21 15 points 5 comments -> Dcmake: A new CMake debugger UI
- 47735927 #30 -> Excellence Is a Habit
- 47736907 #19 54 points 12 comments -> Pope Leo XIV denounces the 'delusion of omnipotence' he says fuels the Iran war
- 47737928 #27 77 points 25 comments -> MiniMax M2.7 Is Now Open Source
- 47735810 #26 54 points 37 comments -> Why meaningful days look like nothing while you are living them
- 47738996 #15 21 points 10 comments -> Why weekends are under threat
- 47737563 #10 298 points 493 comments -> AI Will Be Met with Violence, and Nothing Good Will Come of It
- 47742460 #25 16 points 14 comments -> Are sugar substitutes healthier than the real thing?
- 47743553 #18 102 points 20 comments -> Viktor Orbán concedes defeat after 'painful' election result
- 47744614 #30 -> Orban ousted after 16 years as Hungarians flock to pro-EU rival
- 47744050 #11 13 points 5 comments -> Solar panels are creating an unexpected effect by forming rainfall clouds
- 47745355 #14 4 points 0 comments -> Google Externalised the Cost of Renaming Gmail
- 47747330 #18 24 points 10 comments -> Kindle users in uproar re: latest update, old devices now unusable: 'Fuck You '
- 47675444 #21 242 points 42 comments -> Happy Map
- 47749164 #21 14 points 4 comments -> Opus 4.6 hallucinates twice as more today than when it released
- 47750939 #19 35 points 41 comments -> The hottest college major [Computer Science] hit a wall. What happened?
- 47752856 #14 6 points 1 comments -> An Oligarchy of Old People
- 47753656 #5 4 points 2 comments -> Stealthy RCE on Hardened Linux: Noexec and Userland Execution PoC
- 47753446 #19 8 points 3 comments -> Alpine Divorce: A Hike That Ends a Relationship
- 47756081 #21 36 points 10 comments -> Austerity Creates Fascism
- 47757033 #29 20 points 19 comments -> Google has the same AI adoption curve as John Deere
- 47757775 #29 98 points 114 comments -> The tech jobs bust is real. Don't blame AI (yet)
- 47759400 #18 6 points 0 comments -> SnapState - Persistent state for AI agent workflows
- 47757595 #22 27 points 1 comments -> Mathematical Minimalism
- 47760925 #29 133 points 35 comments -> Hacker compromises A16Z-backed phone farm, calling them the 'antichrist'
- 47766960 #2 44 points 12 comments -> AI Will Never Be Ethical or Safe
- 47770330 #7 7 points 1 comments -> The cost of building a workflow editor on React Flow
- 47770400 #18 2 points 2 comments -> Show HN: Sk.illmd.com, a forum for talking about and showing off agent skills
- 47769967 #16 119 points 2 comments -> California ghost-gun bill wants 3D printers to play cop, EFF says
- 47768195 #6 388 points 366 comments -> Spain to expand internet blocks to tennis, golf, movies broadcasting times
- 47779856 #18 28 points 41 comments -> The Deepfake Nudes Crisis in Schools Is Worse Than You Thought
- 47778366 #27 104 points 2 comments -> US national level OS-level age verification bill
- 47783873 #29 32 points 1 comments -> Jury Finds Live Nation Acts as a Monopoly in a Victory for States
- 47784060 #19 9 points 2 comments -> Show HN: Jeeves – TUI for browsing and resuming AI agent sessions
- 47787368 #22 53 points 9 comments -> Amazon AI Cancelling Webcomics
- 47720941 #16 17 points 6 comments -> A Mercury Rover Could Explore the Planet by Sticking to the Terminator
- 47788773 #20 9 points 5 comments -> AWS/GCP too expensive? Cloudexit.pro will help you move to bare-metal
- 47788857 #10 93 points 61 comments -> IPv8 Proposal
- 47791507 #29 47 points 40 comments -> The noise we make is hurting animals. Can we learn to shut up?
- 47793493 #2 163 points 86 comments -> Claude Opus 4.7
- 47795152 #30 9 points 0 comments -> Armed Off-Duty Cop Tried to Incite Violence at a High School Anti-ICE Protest
- 47733059 #4 104 points 37 comments -> PCI Express over Fiber [video]
- 47736331 #6 59 points 24 comments -> TigerBeetle: A Trillion Transactions [video]
- 47738520 #12 92 points 84 comments -> Put your SSH keys in your TPM chip
- 47737041 #18 7 points 0 comments -> How the Roll Function Works (In APL\360 and Its Descendants)
- 47798468 #20 10 points 8 comments -> Five men control AI. Who should control them?