@moiety wow, hard drive capacity sure has increased since last time I looked
I didn't think 5 petabytes would fit on a desk
@ben 1,6 PiB per stack, 200 TiB per disk 😱.
did they have a bunch of smaller hard drives they were writing to in parallel
or did they just have some absurd amount of RAM available?
or was the process longer than I'm imagining? (although writing 200TiB to a hard drive would still take *ages*)
@carbontwelve oh wow I'm an idiot
of *course* they were doing this over multiple nights
for some reason I had it in my head that all the telescopes were recording at once, but that can't possibly be the case because about half of them would have a very large rock between them and the black hole at any given time
Multiple sites and multiple nights help, but how fast the data can be written is still a limiting factor on the image quality. RAID helps by writing to all disks in a stack simultaneously (RAID-0 I believe), but also the first stage in data analysis is an FPGA that splits one stupendous data stream into many that are only enormous and ships them to separate computers, which then write those to disk separately.
@moiety @carbontwelve @ben the storage system is described in paper II, open access. https://iopscience.iop.org/article/10.3847/2041-8213/ab0c96 page 8. on each telescope there are 4 signal chains. each chain is recording at 16Gbps over dual 10gigE writing to 32 HDDs in parallel. So 128 HDDs per telescope in all. 6-10 TB helium filled disks. Total data size of 1-2 PB per telescope, so I guess that's just one telescope's data on the desk.
The data is formatted as VDIF packets, so while it may be split into different files on the disks it can be reconstructed into the full stream.
This is what happens every time I see something cool.
Like when I saw this cutscene where everyone was amazed by the story going on or the graphics and I was just sitting here going "this is clearly a pre-rendered cutscene, and my character appears in it, so they're definitely doing something very sneaky here and I need to know what it is".
@ben so turns out this photo is from 2017. Might not be all of those petabytes
@moiety watching her TED talk right, love that she talks so much about the elephant at the centre of our galaxy
@moiety Margaret Hamilton continues to amaze me. You can't go to the Moon without a computer, and she was in charge of that. Her work on Development Before The Fact is fascinating. And she coined the phrase 'software engineering'. A real pioneer.
The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!