00:13:47  * ednapiranhaquit (Quit: Leaving...)
00:15:04  * brianloveswordsquit (Quit: Computer has gone to sleep.)
00:23:41  <SuperPhly>are there any quick ways to update a single property of a key without reading the whole key out and then writing the whole key back? for instance, I just want to change the timestamp on a key, but not touch/distrub the rest of the object.
00:30:10  <fritzy>not that I'm aware of SuperPhly
00:37:26  * mhernandez1quit (Remote host closed the connection)
00:39:36  <domanic>SuperPhly, no - there is only get and set. leveldb is not aware of json or anything like that - only sql databases can do stuff like this,
00:39:56  <domanic>because they organize the data differently on disk.
00:40:33  * brianloveswordsjoined
00:40:48  * fritzyquit (Remote host closed the connection)
00:47:44  * kid_icarusjoined
00:48:20  <substack>SuperPhly: you can normalize your data for that kind of thing
00:48:47  <substack>and then construct objects from foreign key strings
00:51:08  * brianloveswordsquit (Quit: Computer has gone to sleep.)
00:51:26  <domanic>SuperPhly, but the question we *should* ask you is what are you trying to represent? there are so many ways to do everything when it comes to databases, there may be a better, different way
00:51:52  <SuperPhly>1 sec guys, client on the phone :(
00:52:14  <SuperPhly>what is a foreign key string?
01:00:13  <hij1nx>SuperPhly: when you want to do X, and you think Y is the best way of doing so. Instead of asking about X, you ask about Y. maybe it's an xy problem if that helps to help you clarify what you want to do :)
01:10:11  * mhernandez1joined
01:27:51  <SuperPhly>Ok, I'll have 500,000 records each has a few categories/tags that they will belong to. I'll need to be able to grab records based on tags, categories (with pagination), autocomplete searches by a few fields, and do a bunch of mass updating of records.
01:28:47  <SuperPhly>hij1nx, substack: i'm a n00b with Node.JS and would like to keep everything in an object model, so I can retrieve, change, update, store, etc really easily.
01:29:10  <SuperPhly>Right now, I'm having to update 3-4 hash tables in redis with each action. It's a recipe for inconsistency.
01:30:32  * mcollinajoined
01:30:41  <SuperPhly>I've been writing this app for about a week really. I started off with leveldb, but handling indexes was a pain in the butt. So I switched to redis, but man that's so low level!
01:30:59  <SuperPhly>I haven't tried Couch or Mongo.
01:34:43  * mcollinaquit (Ping timeout: 240 seconds)
01:39:39  <hij1nx>SuperPhly: sounds like your data is highly normalized, with a lot of relationships, why do you want to use a key/value store? (you can have relationships with a key value store quite easily, im just curious)
01:40:27  <kid_icarus>one day i'll have a relationship
01:40:29  <kid_icarus>;_;
01:41:35  * kid_icarusquit (Quit: Leaving)
01:42:02  <SuperPhly>hij1nx: I guess I don't... I have just been wanting to learn.
01:43:58  <domanic>SuperPhly, there are level-* modules that do indexes for you
01:44:13  <SuperPhly>domanic: yeah?
01:44:29  <hij1nx>SuperPhly: ah ;) so, before you get yourself into a situation where you've learned on code that you depend on running, you could try this -> https://github.com/rvagg/levelmeup
01:44:43  <SuperPhly>do you guys have any reading that i could maybe take a look at?
01:44:53  <hij1nx>SuperPhly: ^
01:45:21  <domanic>for example, here is one I wrote that indexes every json property https://github.com/dominictarr/level-search
01:45:45  <domanic>so, you can just stuff a bunch of json objects into your database and search it
01:46:29  <domanic>also check out substack's thing on top of that https://github.com/substack/level-query
01:47:47  <domanic>there are a bunch more here: https://www.npmjs.org/search?q=level%20index
01:51:12  <SuperPhly>domanic: say I have 500k json objects that has anywhere from 2-50 tags. could i create an index of those tags to objects? As in, "Hey DB, give me all the objects (of the 500k) that are in tag x"?
01:51:43  <domanic>SuperPhly, of course!
01:52:00  <SuperPhly>and it would return the objects, possibly paginated?
01:52:16  <SuperPhly>because that was going to be a real pain in the ass to create with redis.
01:52:22  <SuperPhly>or at least it was the way i was thinking
01:52:26  <domanic>as a stream would be best, which is I think how level-search does it
01:53:00  <domanic>you just create an index [tag, doc_id] : ''
01:53:44  <domanic>and then stream through [tag, null] -> [tag, undefined] (using bytewise keys, here)
01:54:06  <SuperPhly>what's the ,null for?
01:54:17  <SuperPhly>and undefined?
01:54:24  <domanic>in bytewise all the values are sorted
01:54:40  <domanic>null is the lowest possible, undefined is the highest
01:54:51  * brianloveswordsjoined
01:54:54  <SuperPhly>ahhh, ok
01:55:02  <SuperPhly>i've done that before, just checking.
01:55:06  <domanic>you could also do [tag, a], [tag, ZZZZZZZ]
01:55:13  <SuperPhly>that's one thing i liked about leveldb
01:55:16  <SuperPhly>right
01:55:22  <SuperPhly>but now we're playing with fire
01:55:27  <domanic>but null, undefined are better
01:55:28  <SuperPhly>:P
01:55:30  <SuperPhly>yeah
01:55:50  <domanic>well, you can always rebuild the index from the raw data
01:56:13  <SuperPhly>how's that done?
01:56:16  <SuperPhly>can that be automated?
01:56:37  <domanic>SuperPhly, also, have you seen level-sublevel? that gives you prehooks which make it really easy to create indexes, or modules that add a sort of index
01:56:40  <SuperPhly>as in, when i add a new object that matches record-* index it
01:56:52  <SuperPhly>haha, you read my mind ;)
01:57:13  <domanic>this is probably the simplest example of that: https://github.com/dominictarr/level-index/blob/master/index.js
01:57:54  <domanic>(oh this module leaves the query bit as an exercise for the reader)
01:59:22  <domanic>level-search is a more full bodied example - the most important part is here: https://github.com/dominictarr/level-search/blob/master/index.js#L31-L40
02:00:27  <SuperPhly>this makes my head hurt...
02:00:42  <SuperPhly>is there a decent job queue system that runs on level?
02:01:51  <SuperPhly>but then we get back to the issue of updating 100 records... :(
02:02:20  <SuperPhly>that's 100 objects read, pulled into memory, changed, then written back :(
02:02:41  <SuperPhly>i see why this database stuff is so difficult to architect.
02:08:11  * brianloveswordsquit (Quit: Computer has gone to sleep.)
02:11:39  * brianloveswordsjoined
02:28:51  <domanic>SuperPhly, so, the simplest is to use prehooks - just insert the indexes when you insert the record
02:29:11  <domanic>this works best if the index doesn't depend on anything other than the value of the record
02:29:22  <domanic>(which seems like it will work in your case)
02:30:05  <SuperPhly>right.
02:30:11  <SuperPhly>prehooks.
02:30:14  <domanic>if you need something async, there is also level-trigger - that gives you a job gauranteed to runs eventually (and will recover after crashes)
02:31:04  <domanic>that is used by my map-reduce module... which you can use for indexes, but to be honest, usually there is a simpler alternative than map reduce when you are doing leveldb.
02:31:23  * mcollinajoined
02:31:40  * brianloveswordsquit (Quit: Computer has gone to sleep.)
02:32:23  <substack>domanic: what about if the indexes need to change?
02:32:44  <domanic>you mean you want to migrate the indexes?
02:32:55  * mhernandez1quit (Remote host closed the connection)
02:33:09  <substack>yes, what if you need to add a new index for existing data or modify how an index gets computed
02:34:06  <domanic>ah - well I think that is the same as adding an index to already existing data, i.e. you have a batch job that runs once and gets the index back in sync.
02:34:49  * mhernandez1joined
02:35:52  * mcollinaquit (Ping timeout: 240 seconds)
02:36:22  * fritzyjoined
02:38:52  * mhernandez1quit (Ping timeout: 240 seconds)
02:46:28  * domanicquit (Ping timeout: 260 seconds)
02:54:08  * dguttmanquit (Quit: dguttman)
02:55:15  * Sorellaquit (Quit: It is tiem!)
02:55:22  * wilmoorejoined
03:01:03  * ednapiranhajoined
03:03:08  * brianloveswordsjoined
03:32:07  * mcollinajoined
03:33:08  * domanicjoined
03:33:43  * mikealjoined
03:36:52  * mcollinaquit (Ping timeout: 260 seconds)
03:44:14  * dguttmanjoined
04:18:30  * fritzyquit (Remote host closed the connection)
04:20:22  * wilmoorequit (Ping timeout: 240 seconds)
04:30:03  * domanicquit (Ping timeout: 256 seconds)
04:32:57  * mcollinajoined
04:37:17  * mcollinaquit (Ping timeout: 245 seconds)
04:45:47  * harethjoined
04:46:35  * fritzyjoined
04:48:26  * dguttmanquit (Quit: dguttman)
04:49:57  <hareth>hello
05:10:04  * fritzyquit (Remote host closed the connection)
05:29:22  * brianloveswordsquit (Ping timeout: 264 seconds)
05:33:46  * mcollinajoined
05:37:02  * harethquit (Quit: hareth)
05:37:16  * domanicjoined
05:38:03  * mcollinaquit (Ping timeout: 256 seconds)
05:42:14  * wilmoorejoined
05:44:19  * harethjoined
05:46:40  * mcollinajoined
05:51:02  * mcollinaquit (Ping timeout: 245 seconds)
06:18:32  * fritzyjoined
06:21:30  * mcollinajoined
06:25:51  * thefoxisjoined
06:26:08  * ednapiranhaquit
06:26:16  * harethquit (Quit: hareth)
06:42:26  * fritzyquit (Remote host closed the connection)
06:49:10  * domanicquit (Ping timeout: 264 seconds)
06:56:03  * domanicjoined
07:43:16  * domanicquit (Ping timeout: 260 seconds)
08:24:05  * wilmoorequit (Ping timeout: 256 seconds)
08:33:28  * thefoxisquit (Quit: Connection closed for inactivity)
09:03:18  * wilmoorejoined
09:20:59  * harethjoined
09:33:35  * harethquit (Quit: hareth)
09:41:52  * wilmoorequit (Ping timeout: 245 seconds)
09:54:47  * juulquit (*.net *.split)
09:54:48  * Pwnnaquit (*.net *.split)
09:55:01  * Pwnnajoined
09:55:31  * juuljoined
10:13:32  * stagasquit (Ping timeout: 245 seconds)
10:31:36  * harethjoined
10:59:39  * kenansulaymanchanged nick to kenan|afk
11:03:31  * kenan|afkchanged nick to kenansulayman
11:03:42  * mhernandez1joined
11:15:22  * harethquit (Quit: hareth)
11:39:37  * mhernandez1quit (Remote host closed the connection)
11:56:46  * harethjoined
12:24:52  * mikealquit (Quit: Leaving.)
12:39:04  * harethquit (Quit: hareth)
12:41:32  * harethjoined
12:42:47  * mhernandez1joined
12:45:46  * mikealjoined
13:03:56  * mikealquit (Quit: Leaving.)
13:06:35  * mikealjoined
13:10:51  * harethquit (Quit: hareth)
13:11:13  * harethjoined
13:18:16  * kenansulaymanchanged nick to kenan|afk
13:21:56  * kenan|afkchanged nick to kenansulayman
13:48:33  * neonstalwartjoined
14:02:11  * harethquit (Quit: hareth)
14:02:30  * mhernandez1quit (Remote host closed the connection)
14:05:19  * tarrudajoined
14:09:55  * harethjoined
14:13:51  * jjmalinajoined
14:15:30  * brianloveswordsjoined
14:25:08  * harethquit (Quit: hareth)
14:36:19  * dguttmanjoined
14:36:32  * harethjoined
14:54:03  * jerrysvjoined
15:01:22  * harethquit (Quit: hareth)
15:13:12  * Sorellajoined
15:14:37  * harethjoined
15:25:26  <hij1nx>rvagg
15:31:32  * dguttmanquit (Quit: dguttman)
15:32:11  * dguttmanjoined
15:33:25  * wilmoorejoined
15:54:05  * dguttmanquit (Quit: dguttman)
15:57:26  * harethquit (Quit: hareth)
16:01:14  * ednapiranhajoined
16:15:34  * tarrudaquit (Quit: WeeChat 0.4.2)
16:16:10  * tarrudajoined
16:24:07  * kid_icarusjoined
16:29:18  * harethjoined
16:30:14  * mcavagejoined
16:44:48  * dguttmanjoined
16:53:36  * mhernandez1joined
17:33:39  * mcavagequit
17:41:28  * mikealquit (Quit: Leaving.)
17:49:25  * jerrysv_joined
17:50:30  * tmcwjoined
17:52:17  * jerrysvquit (Ping timeout: 245 seconds)
17:56:32  * jerrysvjoined
17:59:50  * jerrysv_quit (Ping timeout: 250 seconds)
18:18:21  * brianloveswordsquit (Quit: ["Textual IRC Client: www.textualapp.com"])
18:28:49  * tmcwquit (Remote host closed the connection)
18:29:53  * tmcwjoined
18:37:16  * mhernandez1quit (Remote host closed the connection)
18:40:16  * fritzyjoined
18:48:21  * kid_icaruschanged nick to kid_falafelrus
19:05:55  * brianloveswordsjoined
19:24:38  * tmcwquit
19:29:06  * kevinswiberjoined
19:30:28  * ellelljoined
19:32:33  * mcollinaquit (Remote host closed the connection)
19:33:08  * mcollinajoined
19:33:57  * kid_falafelruschanged nick to kid_icarus
19:37:17  * mcollinaquit (Ping timeout: 245 seconds)
20:01:02  * wilmoorequit (Ping timeout: 245 seconds)
20:03:43  * mcollinajoined
20:10:10  * mcollinaquit (Ping timeout: 264 seconds)
20:11:58  * tarrudaquit (Ping timeout: 255 seconds)
20:16:42  * kevinswiberquit (Remote host closed the connection)
20:17:14  * kevinswiberjoined
20:21:32  * kevinswiberquit (Ping timeout: 245 seconds)
20:27:04  * tarrudajoined
20:38:10  * jjmalinaquit (Read error: Connection reset by peer)
20:38:28  * jjmalinajoined
20:48:15  * domanicjoined
20:56:47  * mcollinajoined
21:02:22  * tarrudaquit (Ping timeout: 255 seconds)
21:19:08  * mhernandez1joined
21:22:19  * tarrudajoined
21:39:16  * domanicquit (Ping timeout: 255 seconds)
21:43:18  * kevinswiberjoined
21:59:44  * domanicjoined
22:12:33  * domanicquit (Ping timeout: 240 seconds)
22:18:02  * harethquit (Ping timeout: 260 seconds)
22:21:19  * neonstalwartquit (Quit: Leaving.)
22:27:16  * kevinswiberquit (Remote host closed the connection)
22:27:54  * kevinswiberjoined
22:31:57  * kevinswiberquit (Ping timeout: 245 seconds)
22:45:11  * mhernandez1quit (Remote host closed the connection)
22:50:01  * tarruda_joined
22:50:47  * mcollinaquit (Remote host closed the connection)
22:51:25  * mcollinajoined
22:51:54  * tarrudaquit (Ping timeout: 250 seconds)
22:55:22  * mcollinaquit (Ping timeout: 240 seconds)
23:10:39  * daviddiasjoined
23:22:19  * mcollinajoined
23:27:42  * mcollinaquit (Ping timeout: 245 seconds)
23:30:13  * ednapiranhaquit (Quit: Leaving...)
23:32:22  * dguttmanquit (Quit: dguttman)
23:33:44  * jerrysvquit (Remote host closed the connection)
23:46:48  * mhernandez1joined
23:59:29  * domanicjoined