00:12:58  * jjmalinaquit (Quit: Textual IRC Client: www.textualapp.com)
00:23:32  * neonstalwartquit (Quit: Leaving.)
00:37:58  * jerrysvjoined
00:42:46  * jerrysvquit (Ping timeout: 264 seconds)
00:53:13  * stagasquit (Ping timeout: 255 seconds)
01:56:53  * tarrudajoined
02:20:42  * jerrysvjoined
02:24:49  * jerrysvquit (Ping timeout: 245 seconds)
02:39:29  * mhernandez1joined
03:26:02  * mhernandez1quit (Remote host closed the connection)
03:51:09  * tarrudaquit (Quit: WeeChat 1.0.1)
05:40:52  * jerrysvjoined
05:46:48  * jerrysvquit (Ping timeout: 265 seconds)
05:49:17  * owen1joined
05:50:24  <owen1>is there a backup to this project (it uses leveldb) - https://github.com/bigboringsystem/bigboringsystem
05:51:09  <owen1>also i noticed that inside the db folder there are a few folders: bans, logins, pins, posts and profile. does it mean that each folder is it's own db?
05:51:30  <owen1>is it a replacement for tables in a relational db? is this a best practice in leveldb lang?
05:51:31  <owen1>land
06:37:12  <Aria>It works reasonably. I've seen sublevel used for the same.
06:37:16  <Aria>Or partitioned key spaces.
06:37:49  <Aria>But separate databases can be turned into separate services a little more easily.
06:43:38  <owen1>Aria: so when i see stuff like this: db = level('./db/posts); db = level('./db/profile); it means multiple databases?
06:44:56  <Aria>Yeah.
06:46:34  <owen1>nice
06:53:35  <owen1>maybe there is a cronjob that backups the db folder? i can't find anything here: https://github.com/rvagg/node-levelup or here https://github.com/rvagg/node-levelup/wiki/Modules
07:22:47  <Aria>It's tricky because it has to be flushed to disk.
07:39:59  <owen1>so is leveldb (in it's current state) is for apps that i don't care about losing it's data?
07:40:23  <owen1>backup is pretty common usecase.
07:44:47  <Aria>Well, you can shut down the app and back it up, or use something that instantaneously snapshots like lvm or zfs.
07:44:57  <Aria>Or you can use something like level-stream and write out a backup copy.
07:45:03  <Aria>Or you can add a logging layer on top.
07:45:24  <Aria>leveldb, like most databases as people actually use them, doesn't require you to fsync everything.
07:45:32  <Aria>So there's always a window of data that can go missing.
07:46:32  * stagasjoined
08:11:02  <owen1>Aria: got it. thanks!
08:11:27  <owen1>Aria: are those different options for backup documented somewhere?
08:12:07  <Aria>Not that I've seen. Just things I've learned as common between all kinds of databases.
08:12:28  <Aria>And most people care more about consistency than missing data(!) in a great many cases.
08:19:07  <owen1>Aria: what does consistency means for leveldb (or any key/value db)
08:19:47  <owen1>i know when it comes to relational db it can be some rule for a field. like 'don't allow spaces in a username field'
08:19:53  <Aria>That order of writes is preserved, that you won't get half-written values.
08:20:09  <Aria>Transactionality stays intact, essentially
08:20:43  <Aria>Leveldb is damn good. Mostly you can get away with being rough with it and just copying files about. But like anything that takes non-zero time to copy, you get inconsistent data if it's busy while you do.
08:24:05  <owen1>when u say 'copying files about' are u talking about another backup option which is something like "scp -r db remotehost:db-backup"
08:24:26  <owen1>while the db is connected to a running app.
08:25:02  <Aria>Yeah.
08:27:42  <owen1>so here are our backup options: 1.scp 2.shutdown app, scp 3.instante snapshot (lvm/zfs) 4.level-stream 5.loggging layer
08:29:22  <Aria>Yep. Sounds about right.
08:32:26  <owen1>and 1. might be a bit risky. i'll google for lvm/zfs. it sounds like something in the OS level.
08:33:10  <Aria>Yeah, it is.
08:33:19  <Aria>(And on windows, volume snapshot services)
08:33:26  <Aria>(On MacOS, I don't know that there's a clean option)
08:38:28  <owen1>my guess that the first 'scaling' issue http://bigboringsystem.com will be too many connection to the node server. that will require accessing leveldb from multiple node apps. i wonder if that is the usecase for multilevel-http. or maybe another approach is to have multiple instances of leveldb using level-replicate and each node process can have it's own leveldb?
08:39:34  <owen1>i guess the question is how many connections can node server hold and how many connections can leveldb hold?
10:29:49  * stagasquit (Ping timeout: 245 seconds)
10:36:57  * jerrysvjoined
10:41:22  * jerrysvquit (Ping timeout: 255 seconds)
12:05:38  * sveisveijoined
12:38:55  * stagasjoined
14:15:52  <Aria>leveldb would be limited by disk bandwidth. Node by number of connections per IP.
15:45:31  * jerrysvjoined
15:49:52  * jerrysvquit (Ping timeout: 240 seconds)
16:01:42  * jerrysvjoined
16:05:57  * jerrysvquit (Ping timeout: 240 seconds)
16:41:23  * mhernandez1joined
16:50:37  * jerrysvjoined
16:57:42  * jerrysvquit
17:59:48  * kenansulltopic: http://logs.nodejs.org/leveldb/latest — http://r.va.gg/2013/11/leveldown-v0.10-managing-gc-in-native-v8-programming.html
18:26:46  * ednapiranhajoined
18:49:39  <rescrv>Aria: leveldb is limited by way more than disk bandwidth. I have never seen a leveldb instance saturate an enterprise SSD for data sizes above 10G unless the data is inserted in sorted order.
18:50:40  <Aria>Struth. Random access will be more confused.
19:23:38  * mhernandez1quit (Remote host closed the connection)
20:26:00  * ednapiranhaquit
20:26:50  * mhernandez1joined
20:56:23  * stagasquit (Ping timeout: 240 seconds)
20:57:00  * ednapiranhajoined
21:21:13  * ednapiranhaquit
21:52:54  * jerrysvjoined
22:09:48  <substack>owen1: https://www.npmjs.com/package/level-party
22:10:25  <substack>level-party is more for the use-case of running other programs like command-line administration tools while a live server is running
22:11:07  <substack>but if the server goes down a new one can juggle the connection handle
22:12:03  <substack>whenever you open a db.createReadStream you get a snapshot anyways
22:12:28  <substack>so if you create a logging layer that just does a db.createReadStream to somewhere else you'll get a snapshot
22:12:35  <substack>and meanwhile new writes can be coming in
23:10:04  * Raynos_changed nick to Raynos
23:25:38  <Aria>Wow. levelparty is some trippy magic.