Slow Fourier Transform

Какая гениальная мысль, я это непременно реализую.

1. Берем какой-нибудь мега-фид. Например полный фид ЖЖ. На этом этапе проводим предварительное фильтрование (например, выделяем все постинги, содержащие русский текст) и складываем все найденное в базу любого типа — лишь бы постинги хранились отдельно. Самый простой способ, на мой взгляд — переводим данные об источнике постинга в заголовок стандартного емыла, что позволит хранить их любым удобным для емыла способом, а главное, упростит следующие пункты обработки.
2. На этом этапе у нас есть Здоровый Массив Невесть Чего. Теперь мы вручную сортируем его на Интересное и Фигню, разбирая на два больших пакета.
3. Оба пакета скармливаем статистическому спам-фильтру, например bogofilter. Один как пакет содержащий заведомый спам, другой — как пакет, содержащий заведомый не-спам.

В результате мы получим фильтр, который потенциально может без четких ключевых слов и критериев отделить интересный нам постинг от неинтересного нам постинга, чисто эвристически. Этот фильтр может эволюционировать штатным порядком как такие фильтры эволюционируют в случае обработки собственно спама (если он ошибся — показываем ему на ошибку пальцем) и перемолоть в поисках потенциально любопытных вещей гору сходно форматированой информации, т.е. любой другой мегафид отличающийся сходным языком и тематикой.

По сути, он будет показывать нам новости, которые мы не догадались поискать сами, потому что не знали что они есть.

update: Так, первоначальная версия для экспериментов есть, можно пробовать.

По логике, наверное проще всего сначала натравить ее на примеры заведомо интересных журналов, взяв их по фидам конкретно-адресно, а затем натравить на общий мегафид и всегда когда на выходе появится что-нибудь неинтересное, откладывать его в сторонку, чтобы учить базу на отрицательном примере.

Вообще с этим надо будет повозиться гораздо тщательнее, идея кажется очень перспективной.

It’s an old one, but I keep forgetting about it:

Talk someone into recording an album the first track on which would be called Track 0 bad.

I thought about this yesterday and now I think I have confirmed that this is all actually possible and even feasibly practical.

People like internet radio, especially if they have the bandwidth for it. Ogg streams at low bit rates like 40-60kbps sound good enough for dancing to them. Now, unlike MP3 streams, for which ID3 tag metadata is a chunk of essentially totally extraneous data which is not part of the stream, Ogg files are containers by definition — they are a format which encapsulates the audio stream encoded by the Vorbis codec. This is what allows OGM video files to actually work. The metadata chunk is an integral part of the Ogg stream. When an Ogg stream is passed through a streaming server like icecast, the metadata chunk remains intact.

This allows us to do some nifty things which haven’t been tried before.

Since we can actually pass metadata through a streaming server, we can embed data for the player to act upon, and all players which don’t know about this data will happily download it and ignore it. That allows us to distribute the songs through internet radio with various interesting things — like icons, lyrics, and most importantly, stepfile data for StepMania!

Embedding Stepmania data like stepfiles, song banners, backgrounds and other junk into Ogg streams appears to be easy enough to do even with a python script. What’s more important is that even transcoding stream generation programs — at least, I’ve successfully tested this with ices — seem to keep song metadata intact when converting the audio data to the specified bitrates and sampling rates.

Which means that we can relatively easily set up a streaming client that will listen to the specified internet radio, buffering the song until it’s done, and if it contains embedded Stepmania data, signal it to add it to the playlist or even play straight away.

Can you say “radio dance nonstop”? :)

Diskettes are definitely going the way of the dodo, and this is good. It’s a very unreliable media and it’s not very useful with today’s file sizes. However, flash memory is still kinda expensive for a lot of applications. You can freely give away a diskette. You can probably do that with a CDR too, though it’s not quite that cheap.

But a flash card usually isn’t. Old, small flash cards do get cheaper by the day, however, as they go cheaper, the profit margin on them diminishes and the retailers stop carrying them, going up in sizes to keep prices above the minimum. This makes using flash cards as media you would give away impractical.

But there is a kind of media that is an absolute price champion when it comes to giving it away — paper. Sure, it’s write only, but if you intend to give it away anyway, why would you need it to be rewritable? The question is how to store information there so that it would be reliably machine-readable.

The answer would be modern barcode standards which allow you to cram relatively high amounts of digital information onto a relatively small surface — for example, DataMatrix ECC200 used in Semacode and readable with a mobile phone’s digital camera.

If it’s possible to fit, say, 30-100 kilobytes of information onto a postcard-sized card with an inkjet printer, it would be quite useful for small documents, texts, various marker and ID files, and possibly many other applications. Reading it would be as simple as scanning it in or taking a digicam picture — and since all scanners I’ve heard of can do at least 150 dpi, and you can’t expect an inkjet to do more than that, even a very old one will do just fine.

There’s lots of advantages to it. No magnetic data loss, practically eternal storage if the right kind of printer is used. Trivial duplication by photocopiers. The only real problem would be having an image input device like a camera or a scanner, and they are slowly becoming widespread enough for this — most offices have at least a scanner, lots of mobile devices come with cameras.

I must experiment with this.

update, years later: What do you know, someone actually did it.