Gajim - 2024-04-12

Hello

is Gajim the same as jabber

avva3814: Gajim is a xmppp client

lovetox pushed 1 commits to branch gajim/master refactor: Rewrite archive database code - https://dev.gajim.org/gajim/gajim/-/commit/fc147adbe0713a28f537ae98556267a59b357b7b

It's happening!!1!

🎉

a

yeah good luck with the migration !

we create a backup of the old db before migration, so it should not be hard to go back *if* it fails for some reason

and please post the report it shows (if it crashes)

lovetox pushed 1 commits to branch gajim/master cfix: Correctly determine nickname in private messages - https://dev.gajim.org/gajim/gajim/-/commit/e8fef3d72d47c667058735f888365a7e720d5b75

my logs are, like 1.8G

how is the migration story? Should I just checkout, pip install -e . --user and ./launch.py ?

yes, but this will take like half an hour

do you normally call pip install?

it crashes before attempting anything apparently > sqlite3.OperationalError: no such table: remote

can you get me a full log?

did it do the backup ?

yep

hold on

Pastebin’s SMART filters have detected potentially offensive or questionable content in your Paste.

SMART uppercase it must be, then

http://tamytro.org/tmp/gajim_new_db_model.txt

so did it create a backup .bak file?

not even

in .local/share/gajim

its weird its like the migration is skipped, can you check the user_version field of your database?

in logs.db?

yes

I see only 4 tables in there

yes but the database has a user_version field

its not in a table

I didn't know that was a thing

do you use a GUI app for looking at the db?

Sqlite browser shows that field for example

or simply execute

PRAGMA user_version

says 8

Oo

Set it back to 7

PRAGMA user_version = 7

and then try again, but gather logs

rom1dep: did you run the storage branch at some point?

I may have accidentally; I have a script that updates gajim and its dependencies but that sometimes breaks when branches diverge (stuff gets force pushed in weird ways)

ok, now I have a GUI telling me that it failed instead 🙂

I've got to go take a phone call, I'll be back later

It crashed on: File "/home/rom1dep/Binaires/Network/gajim/gajim/common/storage/archive/migration.py", line 258, in _process_message_row account_jid = JID.from_string(log_row.account.jid) ^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'jid'

mhm odd, messges in the database that link to no account

lovetox pushed 1 commits to branch gajim/master cfix: Migration: Don’t fail on missing jids - https://dev.gajim.org/gajim/gajim/-/commit/690ed499681d21985e0c05fc396c471f5b6da52b

rom1dep, i pushed a mitigation of that problem

please pull and try again

migration going on 🙂

10%

25%

after migration, quit gajim normally, this can take also a long time, because it will free up space from your database

50%

I have a buffer full of warnings for message not being migrated due to missing jid or account, I'm curious to see how many messages will come up missing 🙂

My db was smaller after migrating. I'm curious about how much space is occupied by the migrated db, rom1dep

(I migrated before that fix)

let me vaccuum the backup for fairness

80%

vacuuming didn't reduce size noticeably

100%

https://upload.tamytro.org/upload/a3849b367f4dd4cedd5ad6f64afa7645b7d4f834/M9CTfOjdyN92MMotBeMyNrzbzh6xirDRb7ChpHvB/3e472291-b652-4e0f-9c63-50067ec9a54e.png

.. ah damn now the same shit in that table

lovetox pushed 1 commits to branch gajim/master cfix: Migration: Don’t fail on missing jids in mam migration - https://dev.gajim.org/gajim/gajim/-/commit/a26ce1180d14d9f5dd98c7bbe100fbeb85fa1d8a

rom1dep, so now try again please

here we go again 🙂

lovetox pushed 1 commits to branch gajim/master cfix: Message: Store resource for private messages - https://dev.gajim.org/gajim/gajim/-/commit/f4336f45edfb06d826ab1902cd2f25f5838c8f44

done, and new crash

https://upload.tamytro.org/upload/a3849b367f4dd4cedd5ad6f64afa7645b7d4f834/xp8zrqxL88hFxFO116h4oPN7aelDv50yf1rn0yH3/6ae9a782-a331-43be-8973-c065c6d1e95d.png

rom1dep, thanks for testing, this is really useful

it isn't much, thanks to you guys for the great work 🙂

that comes of as sarcastic if you say this after the third failed attempt :D

hahaha, common, I know what I'm getting into by running from master :)

and it's not like I paid a lot of money or anything ^^

hm rom1dep can you export me the table "last_archive_message" ✏

sent a url in PM

thanks

I'm trying to fixup my logs in the interim, the last log line I have with a missing account_id dates back to late 2017

(also, I'm keeping the "broken" original db for those migration tests, of course)

but we would not migrate these anyway

so not need to fix it

or do you want to remap it

I want to remap it

what does the account_id value maps to?

I have, like 7 distinct values for it, from 1 to 221

lovetox pushed 1 commits to branch gajim/master cfix: Migration: Make mam state migration more resilient - https://dev.gajim.org/gajim/gajim/-/commit/f1dbec4ff5bb89f8524584841d9827030c180729

to the jids table

should I attempt another migration ? 🙂

it should contain the id in the jids table that is your account jid

yeah i hope your error is now fixed

https://upload.tamytro.org/upload/a3849b367f4dd4cedd5ad6f64afa7645b7d4f834/oF6cwBSKXDi5y2wXLygWSVkAaxBnxljobewdqfgI/d4e49358-b914-45b1-a468-9f52a4b3024c.png

look at that 🙂

great! how long did it take for 1.8gb?

new size is 507M down from 1.8G

about 10min

https://upload.tamytro.org/upload/a3849b367f4dd4cedd5ad6f64afa7645b7d4f834/duYv6S4TEvP5ivYIt8wXMNW3vMfsuiQrGZOrU9MO/bbe7a8a7-6bb3-46e4-bccf-3125ad572a34.png

apparently there might be some dedup issues

hello, I am trying to setup PGP encryption with Gajim, I already have a PGP key and I want to use it, how can I import this key? Also which plugin should I use? PGP or OpenPGP? What is the difference?

only PGP allows you to use a key

check the plugin config dialog

in the plugin config i think there is a option to define a key

rom1dep, weird how that is the only message dupliated

you can delete that one message over the menu

maybe the mam archive state migration is still not perfect somehow

lovetox, thanks! and one last thing, how do i assign keys to other contacts?

if you select the encryption iside the chat

there comes a shield icon

if you click it i think that shows a dialog where you can assign a key

> if you select the encryption iside the chat > there comes a shield icon oh I see it, thank you so much!

is the deduplication considering the same message from different accounts as duplicates?

there is no deduplication

That works too. I'm just lost in the conversation, carry on

the db migration makes me feel uneasy 🙂

I fixed my logs DB and it's still much smaller after migration :)

fjklp last messages appears duplicated ✏

but my message from earlier does not

could it have to do with messages being received while MAM is fetching?

is it reproduceable? otherwise i would say its some kind of hicup with migrating the mam state

hard to tell as such

OTOH, I'm missing a whole bunch of biboumi messages right now

(W) gajim.c.m.message (tamytro.org) Received message without message id

are messages without message id dropped or not displayed in the new implementation, somehow?

I can almost read the full backlog from my terminal, other clients show it, but not gajim :) ✏

i think its the conversation message receipt

not a real message

rom1dep, no reason comes to mind why a biboumi message would not be migrated

this isn't about migration

those are new messages

MAM fetched

but if you have the jid you could look in the old db how the messages look that are not migrated

ah

i.e. new to gajim

you were joined this chat when the last time?

yesterday

1 day should be requested in any case

hmm

and as I said, the messages are requested, and printed in the logs

not in the messagewindow

and the log says not more than the message id stuff?

2024-04-12T23:06:15 (W) gajim.c.m.message (account) Received message without message id 2024-04-12T23:06:15 (W) gajim.c.m.message (account) <message xmlns="jabber:client" from="#room%irc.ircserv.org@blah/nick" type="groupchat" to="me"><body>X said Y</body></message>

every now and then I get a "Received message without text"

but seems unrelated

yeah this is then really a problem

biboumi is a pain

I thought they came to the point of supporting a decently recent/featured MAM:version, but anyhow, this feels like a new behavior because messages from biboumi were properly displayed and archived before

but i just checked and the biboumi instance on jabber.fr has message ids on the messages

can it be that this particular instance you are using is ancient?

it's the one I run myself, it's the most recent or second most recent commit from the repo

ah mom i see

but its not messages in a chat

can be that jabberfr has patches on top of it, though

i see this on the irc network messages

like the irc server sends you stuff that you should register a nick and something like that, and this has no message ids

if you go to view -> XML Console -> Log

thats a new tab you see errors there

and there is the full stanza that has no message id

can you send me one

I can see the logs, how do I relate that to the stanza? Or you mean, I wait for a new log line to appear and catch the stanza at that time?

anyhow, I'm much less worried about those than I am about no biboumi message showing up anymore

would I be able to see from the logs.db whether those used to have message IDs ?

with biboumi messages you mean messages in a channel, or from the irc server?

in a channel

the stanza is printed under the error message

in the Log tab

new messages seem to appear (moreover, with ID), but MAM messages do not

but this can have different reasons

i doubt biboumi does not add a message id

can i join this irc channel you are talking about?

can I trash the room's history and re-fetch it on demand ?

well, in fact, I have the same problem on 2 different chanels on 2 different biboumi instances (so, 4 repros)

im investigating

if I remove a room's history, then leave it, then join it again, would gajim try to fetch some history from MAM?

no

you need delete the row in mam archive state table

let me try that, then

so, I can totally repro that, then

I left the room, cleaned history, cleaned the mam_archive_state, re-joined

I can see the errors in the logs about "received without message id" and nothing printed in the gajim's messagewindow

lovetox pushed 2 commits to branch gajim/master cfix: ChatBanner: Don’t fail if there is no resource set - https://dev.gajim.org/gajim/gajim/-/commit/6fc27014cb1fef51136ff22bd524344429f7cab2 cfix: Message: Accept messages without id - https://dev.gajim.org/gajim/gajim/-/commit/3b5f46b95d41063885800187212ca35b6dc026ef

Calling it a day, thanks for all the bugfixes, gajim seem in a much better shape now 🙂👍