logs.ericbenevides.com is born: compiled notes on the cloning process

Intended as a companion guide to http://ossasepia.com/2019/09/02/ossabot-and-its-flask-of-python-27-on-centos-6/

I recently stood up my own clone of Stanislav's logotron on a traditional1 Gentoo server. Below are some of my compiled notes and scribblings with hopes that it may aid another in the task of standing up their own clone.

Installation and initial setup

Follow the link to loper-os above, grab the patches, seals, your favorite Keccak V-tron, and press to the znc2tmsr_etc.kv.vpatch.

Now, let's walk through the README. We can see we will need to also install the following2, of which I will list the specific versions I used:

emerge -av =dev-lang/python-2.7.153

emerge -av dev-python/requests4

emerge -av =dev-python/flask-0.12.2

emerge -av =dev-python/psycopg-2.7.4

emerge -av =dev-db/postgresql-9.4.23

emerge --config =dev-db/postgresql-9.4.23

Don't forget to start your postgres server!

/etc/init.d/postgresql-9.4 start

Enter your postgres command-line to see that everything is kosher:

PDS2152 /etc/conf.d # su postgres -c "psql"

psql (9.4.23)

Type "help" for help.

postgres=#

As always, refer to the documentation regarding your specific version. However, here's a useful "cheat sheet" of psql commands:

Command Description
\h help
\q quit or close out of output (such as the 'help' output)
\l list databases in this postgres cluster
\c connect to specific database (e.g. "\c nsalog")
\d list relations (including tables) in this database

Now follow the README in order to create the needed users, permissions, and database, as well as get everything up and running for the first time (I suggest performing the first few runs in a test channel!). However, please make note of a few minor corrections/nits:

The nit list:

1) One micro-nit is that the README specifies the "createdb" syntax, when in fact (from the psql command line, at least) you need the "CREATE DATABASE" syntax.

2) You will need to comment-out the bits in the "reader.py" that refer to caching. Here they are:

#from flask.ext.cache import Cache

#cache = Cache(app,config={'CACHE_TYPE': 'simple'})

3) As diana_coman discovered, we will also need to edit the irc() method in "bot.py" to properly set the global variable of "time_last_conn". The bolded line below is what you must add:

# IRCate until we get disconnected

def irc():

global connected

global time_last_conn

# Connect to one among the specified servers, in given priority :

while not connected:

connected = connect_any(Servers, Port)

# Save time of last successful connect

time_last_conn = datetime.now()

A quick how-to on the reverse-proxy

This portion gave me a minor headache, mainly because I'm a n00b. But ultimately, if you are using apache, here is what your vhosts5 might look like for a proper proxy:

<VirtualHost *:80>

ServerName logs.ericbenevides.com

ProxyPass / http://127.0.0.1:5002/

ProxyPassReverse / http://127.0.0.1:5002/

</VirtualHost>

Where the port in the vhosts config above is the same port you set in the nsabot.conf:

# On what port will sit the www logtron

www_port = 5002

As always, make sure you have the proper apache modules installed on your box. In my case, some key places to look and poke for clues on my machine were:

/etc/conf.d/apache2

/etc/apache2/modules.d

/etc/apache2/httpd.conf

/etc/portage/make.conf << Your "APACHE2_MODULES" flags6

Also take a peek in [your server root]/modules7

Filling the database with those sweet sweet loglines

Okay, so everything is functioning well in the test channels and we got the server servin' the new loglines as the bot ate them from IRC. Now it is time to fill in the archives (the lines that this particular bot did not eat). There are two main paths to go for this one:

Have postgres eat a database dump

This is how I filled the majority of my database. HOWEVER, I noticed that the dump from http://logs.nosuchlabs.com/ was from postgres version 10.3, and I was using 9.4. As such I had to comment out a few options that my postgres installation did not recognize. Namely:

-- SET idle_in_transaction_session_timeout = 0;

-- SET row_security = off;

I also had to comment out this entire block, due to both "loglines_ser_seq" already existing, and 9.4 postgres barfing on "AS integer" syntax

-- CREATE SEQUENCE public.loglines_ser_seq

--AS integer

-- START WITH 1

--INCREMENT BY 1

--NO MINVALUE

--NO MAXVALUE

--CACHE 1;

I also had to carefully replace all instances of "nsabot" with my own "ericbot". I did this via nano, and matched specifically on "Owner: nsabot" instead of simply "nsabot". This is because I did not want to replace genuine instances in the logs.

Feed the database via eat_dump.py

For the gaps and missing pieces, I highly recommend the "raw dump" knob that comes with the logotron. Simply head to a working logotron, such as asciilifeform's or diana_coman's, and craft a curl (or whatever) using the following syntax:

http://logs.nosuchlabs.com/log-raw/ossasepia?istart=1001255&iend=1001535

Note that there is a max limit of 500 lines. After that, simply eat that dump via eat_dump.py (this is well documented, so I will refrain from going into detail).

If you're feeling really masochistic, you could alternatively fill in your database via converting old irssi and znc logs to the tmsr format with the converters in logconverters and then feeding THAT into eat_dump.py, but since that work has already been done why bother?

Getting fancy with custom banner images

Want some spiffy custom images and links on your log? Plox to check out the "templates" and "static" directories. The html templates are where you edit your banner 's img src and href links, and the "static" directory is where you put your favicon.ico and banner images.

  1. i.e. not Cuntoo []
  2. on this Gentoo box, I simply emerged them via portage. Remember your USE flags, of course! []
  3. Make sure the right version is being USED! E.g.

    PDS2152 /etc/conf.d # python --version

    Python 3.6.5

    PDS2152 /etc/conf.d # eselect python list

    Available Python interpreters, in order of preference:

    [1]   python3.6

    [2]   python2.7 (fallback)

    PDS2152 /etc/conf.d # eselect python set 2

    PDS2152 /etc/conf.d # eselect python list

    Available Python interpreters, in order of preference:

    [1]   python2.7

    [2]   python3.6

    []

  4. while not explicitly stated in the README, you also need the "requests" python libs []
  5. on this gentoo box, my vhost config is located at /etc/apache2/vhosts.d/00_default_vhost.conf []
  6. mine looks like:

    APACHE2_MODULES="${APACHE2_MODULES} proxy proxy_http log_config log_debug logio"

    Note!!! the "${APACHE2_MODULES}" bit is very important because unlike the normal USE flags, these flags are not an incremental variable. In other words, if you just specified "proxy" it would use ONLY "proxy". Can ya dig it?

    []

  7. for me this was /usr/lib64/apache2/modules []

5 Responses to “logs.ericbenevides.com is born: compiled notes on the cloning process”

  1. Instead of the "comment out..." errata, why not a vpatch?

  2. lobbes says:

    Is it really vpatch worthy, though? Since it is a version mismatch issue and not really a bug?

  3. lobbes says:

    Err, nvm. My above comment referred to the database dump eating. But then I saw your comment in logs http://logs.ericbenevides.com/log/trilema/2019-09-08#1934999. Loox like I was just impatient after the "WARNINGS" came up on my screen and assumed it barfed

  4. Was in re: the cache thing.

  5. [...] sig: http://lobbesblog.com/src/logotron/uptimefix_bye_cache.kv.vpatch.lobbes.sig In response to http://blog.lobbesblog.com/2019/09/logsericbenevidescom-is-born-compiled-notes-on-the-cloning-proces... [...]

Leave a Reply