znc2tmsr_etc.kv.vpatch

vpatch: http://lobbesblog.com/src/logotron/znc2tmsr_etc.kv.vpatch
sig: http://lobbesblog.com/src/logotron/znc2tmsr_etc.kv.vpatch.lobbes.sig


Author's notes:

Results from testing

After pressing the logotron and applying a few fixes to eat_dump.py, I was able to eat the entirety of #eulora's logs.minigame.biz archive into my test database. Said archive was created via the znc2tmsr code, which is contained within this vpatch along with the fixes to eat_dump.py. Note: the full four year archive took about 40 minutes to eat and resulted in 176887 total loglines.

Special notes on uniturd handling:

As a working example, please take note of index 8412771.

In the 'raw' logs, the 'payload' is as follows:

"Or in my case, re-write a cooking book. So ladies and gentlorans, I give you Foxy’s Euloran Cookbook V1.1,"

Now, in the database, the 'non-standard apostrophe'2 is stored as '\u0092':

nsalog=# select payload from loglines where idx = '841277';
payload
-------------------------------------------------------------------------------------------------------------------
"Or in my case, re-write a cooking book. So ladies and gentlorans, I give you Foxy\u0092s Euloran Cookbook V1.1,"

Now, here's where it got funky for me in testing with reader.py. When reading the output in the 'links' browser, the apostrophe displayed as I would expect:

"Or in my case, re-write a cooking book. So ladies and gentlorans, I give you Foxy’s Euloran Cookbook V1.1,"

Buuut, when I tested in 'lynx' and 'chromium', the apostrophe was omitted entirely (with just a space in its stead):

"Or in my case, re-write a cooking book. So ladies and gentlorans, I give you Foxy s Euloran Cookbook V1.1,"

Same results for other lines that dealt with the 92 hex. So, at this point I'm not entirely sure if this is worth dedicating more time to fix, or if this is something folks can live with. Seems like some browsers will read it, and some will not. I'm open to further futzing if it is deemed a necessity, but if not then I submit this patch for signing.

Various misc threads related to the testing and uniturds (links in reverse chronological order):

flask cache deprecation eggogs; workaround

various testing notes from author; prompt from alf to vpatch

lots of uniturd wrestling/info throughout day; phf drops in with his [algo]

moar uniturd wrestling - 2019-08-15

moar uniturd wrestling - 2019-08-14

moar uniturd wrestling - 2019-08-13

  1. 5 lines down in http://logs.minigame.biz/eulora_log_2015_to_2019_debug.txt []
  2. i.e. not 0x27 []

4 Responses to “znc2tmsr_etc.kv.vpatch”

  1. [...] do not use this vpatch as it is a dead leaf. Instead, see: http://blog.lobbesblog.com/2019/08/znc2tmsr_etckvvpatch/ Following Diana Coman's slick and effective bash one-liner for converting irssi logs to a format [...]

  2. The "cache" thing was in there by mistake (holdover from Phuctor skeleton). Currently it is not used in reader.py, and its include statement will be removed in the next vpatch.

    I set about last week to actually implement a cache -- after prod by MP -- only to find that it makes 0 measurable difference (the annoying load delay turned out to be caused by the proxy (nginx) rather than "reader". The latter, without any caching, generates even the "heaviest" log page (on Dulap) in

  3. Grr, comment cut off, here's the rest of it:

    The latter, without any caching, generates even the "heaviest" log page (on Dulap) in LESS-THAN 0.070s; the rest of the interval, from client's POV, is accounted for by, in descending order, the mystery delay, followed by the transmission delay.)

  4. lobbes says:

    Aa okay, this makes sense and now that you mention it I recall that thread in #t re: nginx, but I must've tuned out on the resolution.

    And yeah, while I was not measuring response time on my testing box, to the naked eye it was displaying lines with no noticeable delay, including searching.

Leave a Reply