00:16:14  * AvianFluquit (Remote host closed the connection)
01:03:01  <amin80>is it possible to run mpipe and mtee from normal SDC instances rather than manta compute instances?
01:07:02  <rmustacc>amin80: You can, but I'm not sure if they're as useful outside of the compute environment.
01:07:28  <rmustacc>The tools are all here: https://github.com/joyent/manta-compute-bin
01:08:18  <amin80>great
01:09:11  <amin80>there is no pkgin for it yet, right?
01:09:35  <rmustacc>I'm not sure, but I'd expect not.
01:09:54  <rmustacc>Is there something particular you're hoping to do with them?
01:20:00  * ed209quit (Remote host closed the connection)
01:20:08  * ed209joined
01:24:09  * therealkoopaquit (Remote host closed the connection)
01:24:48  <amin80>i want to tail logs from compute nodes into manta
01:26:38  <rmustacc>So how are you going to chunk up the logs?
01:26:59  <rmustacc>Generally this is done with logadm to rotate the logs and then making the rotated logs available.
01:27:48  <rmustacc>The challenge with something like tail -f | mtee is that you need it will never chunk up an object.
01:29:47  <amin80>what's the suggested way to stream live data to manta?
01:30:14  <amin80>i thought i can do tail -f | mtee, and mtee would handle chunking
01:30:35  <rmustacc>I don't recall if mtee does chunking like that automatically.
01:31:20  <amin80>generally, can I append to an object with HTTP POST or PUT?
01:31:37  <amin80>if that's possible, I can write my own client to chunk up in 1-2k blocks
01:31:52  <rmustacc>There is no appending.
01:32:07  <rmustacc>With the object store you have to always write the entire object.
01:32:56  <rmustacc>So you'll want your own version of mtee that does that kind of chunking.
01:33:16  <amin80>really? real data normally comes in streams, which end up appending really
01:33:54  <amin80>if there is no appending into the object, then technically it's impossible, right?
01:34:03  <rmustacc>If you want to have partial writes, the tradeoffs for CAP change substantially.
01:34:45  <rmustacc>You have to manually discritize your data. Generally with things like logs, folks are doing that already anyways.
01:35:49  <amin80>how, do they store in small split object?
01:35:59  <rmustacc>What do you mean?
01:36:19  <rmustacc>I'm not sure I understand your question.
01:36:33  <amin80>i mean split an stream of log into e.g. 1k object and write thousands of them
01:36:52  <rmustacc>Well, I don't make them that small.
01:37:02  <rmustacc>But the way we do the granularity is on an hourly log rotation basis.
01:37:10  <rmustacc>At least, that's what we're doing for our logs.
01:37:27  <amin80>i want to provide near real time access to logs over manta
01:37:58  <rmustacc>Sure, then what I'd do is just write something that works on a time/data basis.
01:38:17  <rmustacc>And then just directories to get down to a day or hour and then files inside of them that represent what you want.
01:39:03  <rmustacc>eg. if I was doing something like a syslog end point that put data into manta, then, I'd probably buffer up to 1-2 Mb of data or some set amount of time, and make that be one object.
01:39:10  <amin80>you mean i write hourly into local disk and rotate to manta at the end of the hour
01:39:36  <rmustacc>Well, I only mentioned the local disk strategy as that's something that we do.
01:39:48  <rmustacc>And folks generally use log rotation, so it's a natural fit.
01:40:04  <amin80>then my API should search two places, real time data in local disk and archive in manta
01:40:13  <rmustacc>You could also do something like write a syslog server that writes locally, chunks, and writes to manta.
01:40:28  <amin80>that's exactly what i'm doing at the moment
01:40:51  <rmustacc>Okay.
01:40:54  <amin80>i wanted to avoid local writes, but looks like it's a must then
01:41:09  <rmustacc>Well, what happens when you have a network outage?
01:41:34  <amin80>i use scribe which stores data in a buffer
01:41:54  <rmustacc>And the local writes aren't a must. Your server can just close one file and start putting to the next. You can certainly stream data into manta.
01:41:59  <amin80>and I want to use an f5/stringray to really cover single node outage
01:42:29  <rmustacc>The issue is that at some point in time you have to say I'm going to stop streaming this file and create another.
01:42:37  <rmustacc>And that stream is a single HTTP PUT.
01:43:13  <amin80>the issue is if the server with 1h local data goes down
01:43:16  <amin80>then it's lost
01:43:32  <rmustacc>Well, you don't have to do a one hour granularity, you can choose whatever you want.
01:43:43  <rmustacc>Or if you're using your syslog server, I don't see why the one hour matters.
01:44:13  <amin80>can you explain last one?
01:44:42  <rmustacc>So, you have some syslog server right?
01:44:49  <amin80>yep
01:44:51  <rmustacc>It receives all the logs as they're generated.
01:44:55  <amin80>yep
01:45:09  <amin80>and forwards to my central log server
01:45:21  <rmustacc>And that central log server should put stuff up into manta right?
01:45:26  <amin80>right
01:46:07  <rmustacc>So that central syslog server just needs to take a given stream of data, make a request, stream some fixed amount of data in, or some fixed amount of time, and then make another request for the next lot.
01:46:36  <rmustacc>Given that you're going across the WAN, I'd suggest buffering to local disk, but you certainly don't have to.
01:46:54  <amin80>i want to write small local files
01:47:05  <amin80>and push to manta when a file is closed
01:47:22  <rmustacc>Okay, don't see anything that wouldn't work with that.
01:47:43  <rmustacc>But, I think the sweet spot for object size, is larger.
01:47:44  <amin80>but if single server dies, some small part of data is lost
01:48:23  <rmustacc>That's a property of your internal logging system, not manta though right?
01:48:36  <amin80>maybe i can write a job to aggregate small hourly object into daily/monthly at the end of the dayt
01:48:47  <amin80>right
01:49:05  <rmustacc>Sure, you could.
01:49:25  <amin80>regarding file rotation
01:50:00  <amin80>does it sound strange, if i capture d-trace i-node events of fs-close of the syslog PID?
01:50:14  <amin80>if I don't want to change the code the syslog to send me events directly?
01:50:37  <rmustacc>That's not the first approach I'd take.
01:50:49  <rmustacc>I'd use event ports and use file change notifications on the directory.
01:50:51  <amin80>happy to know what you'd do
01:51:12  <rmustacc>But it all depends on what exactly the setup you have with your syslog daemon is.
01:51:13  <amin80>event ports? don't know what it's
01:51:54  <rmustacc>https://www.illumos.org/man/port_create
01:52:06  <rmustacc>It's a C API, similar to kqueue and epoll/inotify.
01:52:24  <amin80>this is awesome
01:52:42  * abraxasjoined
01:53:17  <rmustacc>Ideally you'd want to basically have your syslog code stage the file and then move it into some target directory for upload.
01:53:44  <amin80>and another process scan the folder ?
01:54:19  <rmustacc>Well, you said you didn't want to change your syslog stuff, so has to be.
01:54:38  * therealkoopajoined
01:55:10  <amin80>my log server already supports rotation to another folder
01:55:37  <amin80>over that folder, then i can use port_create or simply do cron scan
01:56:11  <amin80>sound's logical?
01:56:11  <rmustacc>Sure.
01:56:27  <amin80>my only concern is that now I have to look for data in 2 places
01:56:32  <amin80>local disk + manta
01:56:47  <amin80>suppose I have a dashboard that people can grep a text in log
01:57:13  * abraxasquit (Ping timeout: 245 seconds)
01:57:14  <amin80>then I should send this to both places, which is a bit of pain. but that's how manta is designed i would say
01:57:15  <rmustacc>I thought you had basically turned down the interval so data would make into manta every minute or so.
01:57:45  <amin80>right
01:58:17  <amin80>and i'm eagerly waiting for LDAP based access control over manta
01:58:24  <amin80>i think that's in plan
01:59:16  <amin80>do you know by any chance, how that's gonna be. separate pub keys or a single pub key with different user names?
01:59:35  <rmustacc>I don't remember the specifics, sorry.
01:59:38  * therealkoopaquit (Ping timeout: 264 seconds)
01:59:45  <amin80>that's all right
01:59:49  <amin80>thanks for your time
02:05:14  * notmattjoined
02:13:38  * amin80quit (Quit: amin80)
02:33:45  * notmattquit (Remote host closed the connection)
02:34:18  * notmattjoined
02:39:13  * notmattquit (Ping timeout: 272 seconds)
02:44:54  <mjn>browsing the docs, I see this: "Joyent Manta Storage Service ... bounds the number of objects in a single directory so that list operations can be performed efficiently."
02:45:11  <mjn>but unless i'm blind it doesn't give a number for the bound
02:45:26  <mjn>is there a rule of thumb for what to keep a dir to? a few hundred objects? a few thousand?
02:55:58  * therealkoopajoined
03:00:08  * therealkoopaquit (Ping timeout: 245 seconds)
03:01:08  * sintaxijoined
03:32:20  * abraxasjoined
03:37:29  * abraxasquit (Ping timeout: 272 seconds)
03:42:20  * abraxasjoined
03:44:09  * abraxas_joined
03:44:21  * abraxasquit (Read error: Connection reset by peer)
03:48:53  * abraxas_quit (Ping timeout: 248 seconds)
03:56:35  * therealkoopajoined
03:58:53  * ghostbarquit (Remote host closed the connection)
04:00:11  * ghostbarjoined
04:01:19  * therealkoopaquit (Ping timeout: 260 seconds)
04:10:05  * sintaxiquit (Quit: sintaxi)
04:58:39  * therealkoopajoined
05:02:22  * therealkoopaquit (Ping timeout: 245 seconds)
05:45:11  * abraxasjoined
05:50:00  * abraxasquit (Ping timeout: 252 seconds)
05:58:17  * therealkoopajoined
06:02:39  * therealkoopaquit (Ping timeout: 252 seconds)
06:07:58  * bsdguruquit (Quit: bsdguru)
06:08:22  * bsdgurujoined
06:45:19  * therealkoopajoined
06:45:37  * techwraithpart ("Palaver http://palaverapp.com/")
06:49:35  * therealkoopaquit (Ping timeout: 252 seconds)
06:58:02  * bsdguruquit (Quit: bsdguru)
07:05:41  * ghostbarquit (Remote host closed the connection)
07:06:11  * ghostbarjoined
07:06:12  * ghostbarquit (Remote host closed the connection)
07:16:14  * notmattjoined
07:30:19  * ghostbarjoined
07:34:51  * marsellquit (Quit: marsell)
07:53:36  * notmattquit (Remote host closed the connection)
07:54:13  * notmattjoined
07:59:06  * notmattquit (Ping timeout: 265 seconds)
08:26:20  * daviddiasjoined
08:53:44  * mamashjoined
08:59:18  * sintaxijoined
08:59:56  * sintaxiquit (Remote host closed the connection)
09:46:55  * abraxasjoined
09:51:50  * abraxasquit (Ping timeout: 264 seconds)
10:40:55  * ghostbarquit (*.net *.split)
10:42:52  * marselljoined
10:46:37  * amin80joined
11:40:17  * therealkoopajoined
11:43:24  * amin80quit (Quit: amin80)
11:44:53  * daviddiasquit (Read error: Connection reset by peer)
11:47:51  * abraxasjoined
11:52:05  * abraxasquit (Ping timeout: 245 seconds)
13:39:31  * therealkoopaquit (Remote host closed the connection)
13:48:43  * abraxasjoined
13:53:22  * abraxasquit (Ping timeout: 265 seconds)
14:11:20  * therealkoopajoined
14:16:31  * therealkoopaquit (Ping timeout: 272 seconds)
14:43:50  * AvianFlujoined
15:30:16  * mamashpart
15:37:25  * AvianFluquit (Remote host closed the connection)
15:41:05  * AvianFlujoined
15:41:36  * therealkoopajoined
15:46:12  * therealkoopaquit (Ping timeout: 252 seconds)
16:00:33  * AvianFluquit (Remote host closed the connection)
16:04:30  * daviddiasjoined
16:20:04  * therealkoopajoined
16:24:37  * therealkoopaquit (Ping timeout: 248 seconds)
16:31:05  * AvianFlujoined
16:33:35  * AvianFluquit (Remote host closed the connection)
16:33:56  * AvianFlujoined
16:42:13  * AvianFluquit (Remote host closed the connection)
16:43:17  * overrapart
16:43:24  * overrajoined
17:16:10  * therealkoopajoined
17:54:19  * therealkoopaquit (Remote host closed the connection)
18:25:02  * therealkoopajoined
18:29:38  * therealkoopaquit (Ping timeout: 264 seconds)
19:11:39  * AvianFlujoined
19:15:40  * therealkoopajoined
19:19:43  * therealkoopaquit (Ping timeout: 245 seconds)
19:40:40  * therealkoopajoined
19:51:26  * abraxasjoined
19:56:07  * abraxasquit (Ping timeout: 260 seconds)
20:10:50  * therealkoopaquit (Remote host closed the connection)
20:12:28  * mamashjoined
20:27:39  * therealkoopajoined
20:55:08  * AvianFluquit (Remote host closed the connection)
20:57:33  * AvianFlujoined
21:03:00  * therealkoopaquit (Remote host closed the connection)
21:13:11  * mamashpart
21:20:30  * therealkoopajoined
21:22:46  * amin80joined
21:24:38  * mamashjoined
21:25:33  * therealkoopaquit (Ping timeout: 252 seconds)
21:27:29  * amin80quit (Ping timeout: 252 seconds)
21:35:50  * mamashpart
21:41:34  * therealkoopajoined
21:48:45  * therealkoopaquit (Ping timeout: 252 seconds)
22:29:43  * daviddiasquit (Ping timeout: 245 seconds)
22:30:30  * daviddiasjoined
22:32:24  * daviddiasquit (Remote host closed the connection)
22:32:59  * daviddiasjoined
22:34:21  * daviddiasquit (Read error: Operation timed out)
22:45:26  * therealkoopajoined
22:49:41  * therealkoopaquit (Ping timeout: 248 seconds)
23:53:05  * abraxasjoined
23:57:30  * abraxasquit (Ping timeout: 245 seconds)