"Timestamp Speaker Transcript" "00:03.61 greybeards And now it is my pleasure to introduce Tadlabeck Usa Cto of Iron air. So tad Why don't you tell us a little bit about yourself and what's going on in ione air." "00:12.38 Tad Lebeck Yeah" so nice to meet you ray um yeah ioneer is you know 1 of the companies solving that persistent data problem in Kubernetes you know for stateful applications. But um we take a very different approach to it because. You know the kubernetes value prop is really around being able to run your applications anywhere that makes sense and moving them pretty easily but stateful applications really kind of drag that down you get anchored to where your data is you know a lot of people call that data gravity. Um what we've done. Is a unique thing where we're able to move data across space and time and so it sounds really kind of great doesn't it. But so it yeah well you know and and we do kind of bend the laws of physics with this too because you know like. "00:54.35 greybeards Um" yeah it's a cute trick if you can do it I'll say that much. "01:08.35 Tad Lebeck Ah" across time is pretty straightforward. You know we've seen this with other products in the past where you have continuous data protection because our architecture um the the way the storage system works is that we're able to dial back time to 1 second increments so you know for instance if you're running a ah database new. Dropped a table space by accident shame on you but instead of having to go back to the most recent snapshot you can simply say you know what? I'm just Goingnna dial it back to the second before I did that and boom you're back up and running again. Um and that's a really clever thing so you don't have to think about doing snapshots is just automatically done as you're operating your data. "01:45.00 greybeards That's nice. There's always limitations on how much that of that you can do and stuff like that and the change rate and things of that nature but well I guess we'll get into them of that stuff later." "01:48.22 Tad Lebeck Um" yeah absolutely yeah and then the other thing where it really gets kind of crazy is bill move it across space and you know we we have this great demo where we and we take a a mongo database running in new york. And we teleport it over to dublin ireland and you know Mongo has a way to you know say replicate that data over there and then bring it up so you can actually have remote access over there and so what we do in the demo is we start the Mongo process for that and then we start the ioneer way of doing that and. We're able to get that data up and accessible within forty seconds. It takes mongo a little bit longer to kind of get itself initialized but we run queries on the new york side and then when it's up and available on the Dublin side. We do the same queries. You can see the data is there and. We're able to get that up and going in order of Magnitude faster than the mongol use case. Yeah yeah so you know. "02:43.21 greybeards Ah" an order a man faster That's very impressive. Are you replicating before or or is this something that's done the instant you say go ahead and do this. "02:54.76 Tad Lebeck Well so so it has to how the architecture works and so we don't you know the traditional data addressing for for for storage. Is you know you have a volume and a volume offset and you kind of you know in your persistent volume. That's how you access it. We present that through the container storage interface but beneath this. Underneath the Covers. We actually have a way of naming the data we you know we we create a unique name for it and when we teleport that across we send over that that that library of of names so that the the remote site can start accessing it things that are available. In ah in a globally deducped data store available immediately things that aren't they can reach across and pull them over so you can start accessing the data you have longer latencies to start with all rights are local so it's up and running as soon as you're able to get that that that library or that directory at near. Yeah." "03:45.21 greybeards The metadata the defining the data sitting in the persistent volumes." "03:50.62 Tad Lebeck Exactly and so that allows us to really create. Um" as many copies as you want without any replication necessary because you know the the data is just referenced and if it's in the same cluster.. There's no actual.. There's no copy on write per se it's just simply we write New data. Way Our our data store works is. It's immutable. So every write is is you know itemdempottent that we we store that and so ah another write will generate a new hash and create a new name for it and they'll look up for these things where um that you know from the container storage Interface. Includes the volume and the offset and as well as the timestamp and so we can using that you look up in the metadata server the the actual unique name for it. Anybody who hases to that same value will get the same reference to that. Um and so that allows us to really kind of automatically dedu things because there's no need to rewrite it again. "04:36.30 greybeards Um." "04:48.90 Tad Lebeck Um" and then you know as you do writes if it exists already then you don't need to even write it. Um. "04:54.95 greybeards Typically you know deduplication occurs on some sort of a boundary. It can be at the file level. It can be at a block level. It's unclear to me is I own near a block device or a file device." "05:05.39 Tad Lebeck What so you know in in the Kubernetes world persistent volumes are are block devices. That's what we present and that you know Kubernetes lays a file system on top of that so we present a block device into the kubernetes you know storage interface." "05:13.61 greybeards Okay" and it's and it's read writeable by multiple containers or is it only a single container readw write. "05:23.27 Tad Lebeck Well sort of this thing. So So the the usage model on kubernetes is there's not multiple reader writers typically they the the notion is you have your data that you operate on it. Um" and so because of the way our our dedup works you can have. Multiple readers off the same data but they have their own view So when they do their writes. They write to their own namespace. "05:48.82 greybeards Yeah" and that Namespace is a um artifact of I own here. It's not a kubernetes namespace. "05:55.31 Tad Lebeck Well so it's it. You know when I see the namespace. It's really that addressable space that's presented for a persistent volume claim and so it's a kubernetes notion. Um" and because we're we're built for and in Kubernetes you know we're a microservices architecture. Um. "06:04.23 greybeards I got you I got you." "06:14.83 Tad Lebeck That's just what we do. We don't try to do this outside of kubernetes. We're really built for and with kubernetes." "06:18.39 greybeards Right? right? And so the containers let's say in ah operating in a cluster when they're accessing the persistent volume. That's a persistent volume have to be on the same worker node as the container is executing it just has to be in the cluster someplace." "06:28.90 Tad Lebeck No" no no it is so it has to has to be in the cluster or you know when we do teleport outside of clusters. It's a special use case because you have to be able to but inside the cluster. "06:39.27 greybeards Bright." "06:44.50 Tad Lebeck There's the notion of those those those nodes that contain data and there can be other worker nodes who are accessing data and they don't have to all be the same." "06:49.62 greybeards Um" and the and the the worker Knowdes that contain data can also contain actually run containers as well and they're running Obviously the the storage container service. Whatever that turns out to be. "06:54.95 Tad Lebeck Absolutely" that's right? So we we you know we we run in Kubernetes as these microservices and if if kubernetes decides to run other containers in that same network or node that's that's up to them. "07:10.95 greybeards And they um" write so the item potency of of the data means that you know if I do a write. It's it's it's forever so I can go back to the first write to a block. As well as the last right to the to you know the the block and all that's being maintained in and ioneer storage and metadata. "07:37.80 Tad Lebeck Yeah" exactly and now with with caveats right? because as you mentioned before if you if you run this long enough. You'll just consume all time and space. So we do have the ability to say you know look um preserve this for some time period and you can dial that between you know. "07:55.97 greybeards So the continuous data protection you can apply and say for this persistent viome save it for our last 2 days or something like that right right." "07:56.52 Tad Lebeck Hours minutes days months." "08:02.13 Tad Lebeck How how long how long term you retain that so you can you know what window? Do you want to dial back into and and then we also have the ability to to name um" certain points in there so we can bookmark places and then those are preserved until you actually manually get rid of them. So. "08:16.20 greybeards So like if a ransomware attack occurred you could go say okay" the second before a ransoware attack the good volume something like that. "08:24.47 Tad Lebeck Absolutely" that's exactly 1 of the use cases it. So so ioneer is is the is the reboot of a company that was called reduxio in the past and so they were doing tin rap software um and you know as the market moved away from enterprise appliances. "08:34.30 greybeards Um." "08:44.46 Tad Lebeck They pivoted into the kubernetes space and so that ability they actually have great customer experience with um I think it was a police department on the East Coast where they actually had a ransoware attack and they were able to just dial back time and recover without any implications on it. So." "08:57.70 greybeards Yeah" yeah yeah so a couple things um container to do containers and pre persistent viomes. Do they have affinity to can they have establish affinity to a node and things like that. "09:10.77 Tad Lebeck Is so so what we do in this environment is we actually utilize the ephemeral discs on the on the nodes and and we pull that together as a storage and present that as logical volumes of the environment. So the nodes that have." "09:20.14 greybeards Um." "09:29.24 Tad Lebeck Ephemeral drives that you've allocated as data storage nodes um will present that and so if kubernetes wants to do you know hints towards people running with ah closer to the data they can do that. But that's not something that we would actually manage." "09:41.95 greybeards And as far as data protection for the data are using like raid structures or or erasure protection or replication or." "09:51.96 Tad Lebeck So so the the current version is using a threeway mirror. We sort of reach a porm and we have on the roadmap to do erasure encoding. Um" as we scale that out because you know these clusters can have. "09:57.98 greybeards Okay." "10:09.70 Tad Lebeck More than 3 notdees as a matter of fact" many of the clusters have a lot of nodees and we only require through a mirroring across that. So as we look at doing erasure encoding you want to do larger writes there so you have to be a little clever about taking small writes and grouping so you get effective stripe writing across the erasure encoding. "10:26.28 greybeards And and the the data is effectively I I would call it log structure on on the ephemeral discs across all the solutions or is it." "10:27.41 Tad Lebeck Volumes." "10:37.74 Tad Lebeck Well no" we don't So we're not yeah it's not log structured so that's that's we're a little bit different on that in the sense that the way that we do It is you know we we were able to do this this threeway mirroring for quorum and making sure that we have availability across that but the metadata server is really the. "10:43.75 greybeards Oh." "10:57.19 Tad Lebeck The the value on that and so there's no log replay or anything like that. We simply um" you know you you query based on the timestamp you want and you get back The the blocks that you want for that then you navigate your file system accordingly. So there's no long replay or anything like that. "11:13.43 greybeards Right? right? And so the metadata server becomes a crucial data integrity aspect of the solution. How is that protected." "11:22.18 Tad Lebeck Yeah" so so the the metadata server actually uses ah its own um threeway mirroring and it has a ah another in distinct from the user data way of storing that data because you don't want to have rely on the same thing. So the metadata scripter actually does that then we. "11:38.14 greybeards Right? right? right? And so you mentioned um I don't if it's data migration or data replication across Clusters and so in and that scenario you would replicate the metadata server for that." "11:41.99 Tad Lebeck Protect that independently." "11:57.43 Tad Lebeck Right? So witness only only for the volumes that you want to move and so you yeah so we don't we don't need to drag the whole thing over. We just simply say hey this is a volume you want to bring that over here Boom more." "11:58.30 greybeards Named volume or for all all the data sitting on the cluster I see and then." "12:11.50 greybeards And then so at the at the Target cluster when accessing that named volume if the data is not there. It goes out to the originating cluster I guess to get the data but are you moving the data in the background So that in fact." "12:24.52 Tad Lebeck That's correct." "12:27.20 greybeards The data would be replicated in both places or how do you control that." "12:29.90 Tad Lebeck So So we we we just do the heat maps bring things Over. We don't bring everything over um and then because of this this this global dedub store in each Cluster. Um. The other cluster Iss going to have its own dedup and any data that hashes to the same name.. There's no need to copy it over so we don't start copying everything over because some vote may exist there already from other um volumes and so we simply do it on on reference." "13:00.80 greybeards So you so you kind of merge the data between Clusters is and on a heat map basis and and so you you know" let's say I've got this via name Ray on on my cluster at home my crypto clusterster and I want to. "13:06.22 Tad Lebeck Um" it's so no we don't want to. "13:19.59 greybeards I want to replicate it to the tad cluster sitting in I don't know east coast someplace." "13:26.35 Tad Lebeck So So because we're the the data is you know we we do a secure hash on that we we would have a name on it if you have data that hash to that same value. It's the same data so we don't have to copy that over it's that exists in your system already. It just happens to have the same. You know" unique global unique name for it and so for those they just exist already because we name the data based on the contents. Um and if you happen to have data the same content. You know a string of Zeros For instance um we don't need to copy that over right? It exists already in there. So. So We're not really merging the Data. We're only simply saying look if you already got data that exists with that same globally unique Name. We don't need to copy it. "14:11.00 greybeards Okay" but but you know so I got this viome on my crypto server called e t h 1 or chia 12 or something like that and I want to I want to replicate it over to the tad cluster on the East coast "14:18.50 Tad Lebeck Sure." "14:25.54 greybeards And so the metadata gets merged right? I mean so this metadata that's associated with shia 12 is a list of hashes. Let's say for the data blocks right? is now how this works." "14:31.57 Tad Lebeck Yeah" yeah. That if you you've got the right model. "14:39.84 greybeards And and then that metadata is going to be copied over to the tad cluster and now chia twelve as a volume sort of simulated but exists on the tad cluster now as you access the data. Are you generating the heat map at that point or do you already have a heat map from the ray cluster that you're copying over." "14:57.27 Tad Lebeck We have the heat map from the the access patterns from the the cluster that was that you were teleporting across." "15:01.42 greybeards Also so you'll and you'll you'll copy that data over over time so that it makes sure it's available at the the tad cluster huh." "15:07.40 Tad Lebeck Um" yeah the data will need a copy exactly. So so it it allows you to teleport things across like that now those are unique cases probably that the more germane use case that we should probably zero in more on is around you know think of your your normal cicd pipeline. You've got a team of you know engineers working on this and you. You you've got a ah a database that you've cleaned for the development team to use. They typically you know have to go through and make a copy of that data. Um as they go through their you know their their devop right line right? And so so as they copy that that takes time and then they. "15:45.41 greybeards Dev test kind of thing. Yeah" yeah. "15:52.80 Tad Lebeck They they move it through the different you know integration test deploy to Dev deploy to Qa and as they go through that they're copying data across the pipelines with us. They simply just get another name for it and they start using it and when they do hit an error instead of having to create a new copy of it. They can just simply say reset it back to the time. So I have consistent data to test against again. Um" so you know if you look at that that that pipeline goes from hours down to minutes because you're not cropping into data. Yeah and so and and more importantly you can now have parallel streams going because. "16:12.62 greybeards And." "16:20.60 greybeards Because you're not copying data anymore. Yeah yeah." "16:29.52 Tad Lebeck You don't pay a price for that extra data because mostly it's reading and what they do write is what they get separately. But now you can actually have you know multiple streams of developers working off that same dataset without interfering with each other and then resetting and moving along so you really eliminate the data bottleneck and you really let. Ah" pipeline goal much faster and that's a use case. That's more typical right? This notion of teleporting across is is unique and it's is useful. So some cases but that's not the the main use cases really around this devops pipeline you know acceleration. "16:48.40 greybeards Ah." "17:02.17 greybeards And and that's because you're not really copying data to go from dev to test to q a or something like that. You're just copying Metadata I guess right." "17:08.14 Tad Lebeck You created what you in those cases. It's all in the same cluster so you simply say here. Give me you know I want my own I want my own copy of that you get the name of it and then you you access the same things. But now when you do rice you create your own your own right? patterns." "17:22.90 greybeards Right? right? right? right? So why? Why do you think that stateful containers are becoming more important in the Kubernetes world I mean it's always been stateless plus the databases have been outside the kubernetes environment. But." "17:37.20 Tad Lebeck Yeah." "17:39.11 greybeards Something's happened in the last couple of years to make stateful containers be more interesting." "17:41.76 Tad Lebeck You know I It's always been there but I think that you know the the initial push was let's get stateful. Let's get stateless right? Let's just go do those? Um" and yeah. You'd look at the big financial institutions. They've got you know 20000 programmers. They've got all these applications who developed using various databases. They don't want to rewrite those to target them to a database service in the Cloud environment. They want to bring them along because the guys who wrote those things have moved on but they want to bring it into this environment. And so what you're seeing now is as people bring stuff over the it's It's more the refactoring right? They they kind of bring it over the database comes over as a service but then they refactor the application to a microservices but they don't need to have to retarget it. Um and so there's a lot of you know. "18:19.91 greybeards It's a lift and shift kind of thing right." "18:28.98 greybeards I say." "18:36.48 Tad Lebeck That's what's pulling things now whereas before it was like you know the first wave was we're going to develop it in the cloud for the cloud and we're going to use all the services there. But as you start to move towards people thinking more seriously about Multi -cloud that's harder to do because to retarget to another cloud than you have to retarget to that other database services if you. Bring your data with you. You don't have to do that." "18:56.61 greybeards Ah" so what's a typical installation look like for ironer these days from from a data size perspective and I'm not I Guess the not the replicated data but but the actual data. "19:09.22 Tad Lebeck That's ah yeah" that that the answer is always the 1 you hate which is it depends um because you know the the example 1 of our customers is using um citus. "19:15.85 greybeards F." "19:26.80 Tad Lebeck Ah" you know the progress the postgres database and and and for them. It's a fairly large database and what they love about us is because we have that availability when sinus would go down before and then have to wait for it to rebuild everything um and with us because of the availability in the in the mirroring. We're able to have them back up in Minutes instead of hours and so the size I don't know the exact size of it's a fairly substantial thing because they're collecting all this security information. Um and then you know you look at the devops cases in those cases. It's really hard because for some of them. It's production data and those can get fairly big but. Most databases in this space. Don't get huge right? I mean you don't see Multi -terabyte you know you know it's if they're usually you know smaller databases that are built around for that application. Specificcific use case. Um. "20:07.00 greybeards Right." "20:16.30 greybeards Right? right? right." "20:19.13 Tad Lebeck So yeah" you're not seeing that the s a p oracles moving into this space If what we haven't seen him yet as I yeah here we. "20:21.96 greybeards Yet" Let's hope not maybe I don't I don't know that's a different discussion. So you mentioned high availability. How would you characterize Ionee solution is it a highly available storage environment or not. "20:37.50 Tad Lebeck Yeah" so so if you think about you know in the in the Kubernetes cluster environment because we've got this data mirrored across you know a cluster of machines as as nodes fail especially in the public cloud as they go away. We can steep keep running because we can still have quorum. If you blow the whole thing out then you have to kind of rebuild it. But um and that's that's a cluster failure as opposed to a node failure node failures are much more frequent and in that case we provide. You know it just keeps running and then we can you know we can keep moving along on that so it it is a high availability solution. "21:07.30 greybeards And." "21:16.34 Tad Lebeck Targeted at you know persistent workloads in this environment." "21:17.38 greybeards Right? right? right? right? So the metadata becomes extremely important during high availability solutions. Are you backing up the metadatas or some sort of I mean obviously the triple redundancy helps but are you replicating it." "21:25.45 Tad Lebeck 1" "21:35.10 Tad Lebeck So so in the on-prem solution. It's it's you know you you back up your systems how you back them up and we we provide you to do that in the Cloud we persisted off to you know that the the elastic block storage and aws for instance and then you can you can snapchat it if you want." "21:35.13 greybeards Someplace else or." "21:54.70 Tad Lebeck We haven't integrated with the other backup products as of yet. But that's the thing to start looking at it and then on our roadmap for future work is really this ability to tier across different storage types because our microservices architecture allows us to add new storage nodes in the background so we could have. You know the the the ephemeral drives we can have ebs volumes that we can even have s 3 and then the ability to move data across those as we need to. But that's road maps in the future. It's not in the product today." "22:24.16 greybeards And so so today would you be using ssd storage I mean does the storage have to be a the same capacity across all the storage nodes I mean the storage disks and b does it have to be the same type. Could you mix ssds and disk drive. For instance." "22:39.97 Tad Lebeck You know? Yeah" so so the the use case that we've seen mostly people have stopped using hard drives except for really large volume storage and everybody just kind of solves their I o probably with Ssd now because the price points are so low. "22:42.29 greybeards And I'm not sure what the ebs terms are for those." "22:51.87 greybeards Um." "22:59.28 Tad Lebeck So we haven't really tried to mix and match ssd and hcd because that would just be the latencies are so far off so we really you know on the ephemeral drives we work with ssds typically with nvmme if it's available because that's the best performance we can get out of it and then we're working on. A blended thing where we'll use that with a G p 2 or whatever the storage you want to use on the ebs farms and the backside." "23:21.33 greybeards And you mentioned storage sharinging as ah as a future deliverable What other future deliverables. Are you guys working on I know if you want to talk about and that's the question." "23:29.51 Tad Lebeck So the yeah I don't want to get too much into the roadmap right now because I think that you know I don't want to get ahead of myself here I." "23:40.27 greybeards All right" right? right? So how many? ah so about how many customers I mean I know she have a free download free online thing but I didn't see anything about the pricing. So maybe you could talk a little bit about the cost of such a solution. "23:53.52 Tad Lebeck Yeah" so we we typically price out like the other persistent storage solutions for kubernetes which is on a per node cost basis and equal. "24:03.73 greybeards So this is per node in the cluster whether they're data or or worker nodes or does per data node. Okay so I could you know I could potentially have a data node with 24 ssds and and if I had you know on-prem or something like that. That's." "24:07.70 Tad Lebeck Now the per per per data no per data. Not yeah because you know we we the the value we get to working on." "24:23.60 greybeards The same price as 1 with 3 That's interesting." "24:23.85 Tad Lebeck Yeah" yeah yeah we we bented around the idea of capacity based pricing and I think it's just as you look at where the market's going. It's It's really you know this edge to private to public Cloud and we want a pricing model that worked across those and. We do price differently for the on-prem versus the Cloud because the Cloud things typically but it's still a per node basis. "24:50.30 greybeards And I know I noticed on your on your website you mentioned edge quite a lot I hadn't considered persistent volumes being relatively useful on the edge. Are you seeing a lot of call for that." "25:00.49 Tad Lebeck So so we're we're getting into good conversations on it now and I don't want to be too specific on it. But if you look at the whole um evolution of 5 g it's all kubernetes based and the as we start to see how." "25:13.00 greybeards Right." "25:19.38 Tad Lebeck Ah" carriers are building out these micro data centers if you will themselves well they' were actually putting little data centers out there as well and then you look at some of the use cases where they're actually doing remote 1 1 company I had talked to they were ah a mining company out of a scandinavian and. "25:20.36 greybeards Cells or whatever a." "25:38.97 Tad Lebeck When they go into a place they they plop down a little edge data center and then they have a bunch of drones that go up and survey the land and and figure out the mining process for it and that's local data then linked to remote Data. We're not working with them yet. But I Just love their story because they're really. Doing it for completely edged to the public cloud in that whole pipeline because they they have to be able to run independently when they're in these remote locations but they will only get up to a central system because they use that data to optimize their mining processes as they go across." "26:04.67 greybeards Um." "26:12.00 greybeards Right? right? right? So in that in that scenario where you've got this persistent viome really being generated at the edge but you want it to be available at the the core center or something like are on the cloud. Um" would you be pushing more data than just a heat map when you replicate that data I mean you know the heat map is an interesting idea but the data doesn't actually exist in both places right. "26:36.61 Tad Lebeck No" it's right? So so so you're right? and that's well the data exists if it if it if fits the right? but but you're right in this case where you want to move data to the core you have to move the data right? because that's because. "26:46.96 greybeards It's accessible. Yes" but the latency is different right. Right. "26:56.40 Tad Lebeck The other the other case of ironbar is more of remote access coming in. This is really pushing data back up. Um" and that's a roadmap thing for us to look at how we you know replicate data into a core as opposed to the. "26:59.24 greybeards Yeah." "27:10.24 Tad Lebeck Current model which is yeah your data is operating in the cluster and you have you know you want to teleport access method out." "27:13.87 greybeards Right? right? right? right? As far as there's really no limit to how much teleportation or how many different clusters I can teleport some ray volume out to if I want chia 12 or something like that." "27:28.21 Tad Lebeck Yeah" that that's right it. It's really because it's It's a simple naming um system and you basically just yeah yeah. "27:36.26 greybeards And metadata of replication kind of thing. Yeah huh and then so that you mentioned d-dup and you're using some sort of hash is this like a shot to fifty 6 or five twelve on the data or do you talk about that shit." "27:49.68 Tad Lebeck Exactly at our normal so it's a shah hash I don't mean I'm not going to get into the details of but but exactly that right we do a secure hash on the data to come up with a unique name and then the metadata server will map volume offset and timestamp into that secure hash for the Ae for the data that was." "27:59.65 greybeards M." "28:08.10 greybeards Right? right? right? So all snapshot technologies and stuff like use copy on write or copy on. You know you're not doing that because you replicate you're creating if it's new data." "28:08.70 Tad Lebeck You know Index by that." "28:15.39 Tad Lebeck We don't have to do that. Yeah because then you did you just create new data. Yeah" if if it exists or they exist and the name is already there and if it's new data. We just. "28:24.83 greybeards And it's a new entry in the hash table if it's old." "28:33.74 Tad Lebeck Write the new data. So there's no copy on write per se." "28:37.46 greybeards But the metadata has to be updated. Regardless right." "28:41.10 Tad Lebeck Yeah" so the metadata is going to track us so you're focusing in on the right thing and the real unique thing about this product and the the implementation and the and the the beauty of it is is around the metadata and there's a lot of magic inside that in terms of how we do that and a number of patents on that. Around how we're able to you know like the age-old problem of you know writing data and the metadata and you know how do you manage that synchronously because you don't want to end up with a a dangling reference or data that's been working. Yeah and we actually have a unique way of doing that simultaneously and and maintaining that. "29:09.49 greybeards Or something like that. Yeah" yeah yeah and yeah you mentioned earlier that you you can add data nodes to the cluster. Ah. "29:18.29 Tad Lebeck And we've got patents on that." "29:29.40 Tad Lebeck Um" but that's right and you can lose nodes as well. "29:29.41 greybeards On Demand without having to take the system down or bring it back up or anything like that. I Understand the lose nodes but lose nodes is you know they were there all along and they're gone now. Okay" but now but now I want to add another node to this Cluster. So How do you do you? do you spread the information now at this point or and or not. "29:39.53 Tad Lebeck Yeah" so clean Well but but it's kind of the same problem. Yeah. "29:50.57 Tad Lebeck Yeah" you do we do we we we we spread it out. We don't try to balance it completely. But we do start to balance it some so you get you know better resiliency and better utilization different nodes and so you can keep adding them when we spread the data around them. "30:02.84 greybeards Is is the is the data physical address some sort of a hashing mechanism across the node. So I mean I've got a block I want to write to something I I have to decide where in the ephemeral storage or okay." "30:15.22 Tad Lebeck So is correct What what? let's let's talk about the architecture for a little bit I think that will help address this right because because you're you're going right? where the architecture leads you to which is you know the basic model is that there's." "30:22.52 greybeards Go for that." "30:30.78 Tad Lebeck 3 main microservices to the product and the metadata 1 We've talked a lot about there's also the the front end that presents the Csi um" and and we call those the the pictures they basically catch the data from that thing and they send it off to the metadata server to map it and then they go to the persistent storage. Um. Things we call the keepers and the keepers will take care of storing that data and um so when you look at how this works The the thing that interfaces with the csi is the 1 who knows how you're accessing the data and he simply goes to The. The metadata services you know I've got this you know volume Offset Timestamp give me the name for it and then he knows which keep her to go to to ask for it and the keeper then will do the mapping to the actual physical storage. So So they. "31:20.99 greybeards Even though the physical storage might be on some of their data node." "31:25.33 Tad Lebeck No" so so the keeper is the thing that actually owns the data that they live where the data where the storage devices are and so you know what the exceptions of things like it if you have a keeper for for s 3 It's it's going to be the interface s 3 in this case but they're the ones who actually know that they're the ones who map. "31:29.73 greybeards Oh okay." "31:44.86 Tad Lebeck You know" kind of like that name to the physical storage location which is independent. "31:46.60 greybeards But in this case" the physical storage location is at least 3 different physical devices right? and and and does do those physical devices have to be on the same note or can they be spread across different. "31:53.95 Tad Lebeck That's right" the keeper the keeper in that case knows that. "32:02.26 Tad Lebeck Oh no" they have that we want but we want them across separate nodes. So we have better availability. "32:05.50 greybeards Data nodes so they are so a keeper actually can access the data on other nodes than the physical data nodes. It's access. It's on." "32:11.21 Tad Lebeck So it if the sort to keep Yeah" so in in the case of this it. It understands the topology of where the data is store and and and how that works exactly and so so the so the keeper masks all that from everybody else so you don't. "32:21.55 greybeards Right? right." "32:28.86 Tad Lebeck Anyone accessing it doesn't have to understand that the keeper is the 1 who presents that up." "32:30.89 greybeards So the keeper in my mind is ah essentially a block server. It says here's a block I want to access and it goes and finds and gives it back to you or something like that." "32:38.53 Tad Lebeck Yeah" a could it's It's yes exactly So you you come in with the name but say here you know here's the thing I want. Yeah I you yeah I have the name for it. You go give me the data associated with that name and bring it back for me. "32:51.29 greybeards And there's there's effectively a keeper associated with every data node." "32:55.44 Tad Lebeck So there's a keeper associated with every data storage type so you can have keepers for the you know this this threeway mirror dephemeral. You know a keeper for ebs. You can have a keeper for s 3 um and they understand how that data topology is stored." "33:11.96 greybeards So there's 1 keeper per tier I would call it and and and so but the keeper has to be it has to be." "33:16.76 Tad Lebeck Yeah" you can call it it here. "33:25.24 greybeards Replicated across multiple nodes. So in case" 1 goes down it can continue and go on and stuff like that and ah does the keeper become a bottleneck in this scenario if if if I've got flash devices on-prem ah you know and I've got. "33:27.13 Tad Lebeck Right? exactly." "33:39.18 Tad Lebeck A." "33:42.44 greybeards I guess I would have at least 2 keepers if not more right for that." "33:43.83 Tad Lebeck So so so so the way it works is that no the keeper does not become the bottleneck on that because it's simply the the mapping of it. So if you've got um a lot of data nodes you you understand a mapping of that and you can say look I need to go get that data from this spot over here." "34:01.51 greybeards But tad" there's only 1 keeper for that solution right. "34:03.40 Tad Lebeck But no" so so let's let so there's a type of keeper I didn't say I'm sorry that there's not necessarily 1 instance of keeper. There's a type of keeper and there's multiple instances of the keeper. Yes so yes. "34:12.63 greybeards Okay" and so I could scale up to keepers if I thought I needed more I could scale them down if I didn't need more etc etc. "34:22.32 Tad Lebeck Absolutely so because we we built it to be you know" kind of a ah good microservices thing you can scale the the catchers you can scale the metadata service you can scale the keepers to to match your your your workloads. "34:35.65 greybeards And adding a new data node just adds it to a keeper set that it knows about and it can start migrating data if it needs to but in the end it's it's It's just there." "34:45.82 Tad Lebeck Yes." "34:53.11 Tad Lebeck Right? So so you and that's that's all hidden from you. You don't have to think about that right? the keeper just presents it to you and it'll manage so you know I forget what our biggest um cluster is but it's an on date. It's an on-site 1 but it it could scale pretty large right? And so um." "34:54.51 greybeards Right." "35:11.29 greybeards Pretty large petabyte pretty large terabytes." "35:13.10 Tad Lebeck We haven't seen that same problem I think it's terabytes I don't hold you to that 1 I don't know the exact number on it." "35:20.60 greybeards Yeah" yeah yeah so um Kubernetes has become very important to vmware as well. It's ah the whole you know I can't even think of the name now but tanzoo. "35:31.12 Tad Lebeck Um" comes yeah yeah. "35:33.67 greybeards Services and stuff like that. Do you guys operate under any kubernetes cluster or is there certain ones that you work with or." "35:38.62 Tad Lebeck So is it funny you bring up tonzo and they say do you operate under any. So yeah" so so today we work under any kind of um standard kubernetes cluster. So if you abide by that and then you know. So if it's your own operated cluster versus even um a managed 1 as long as it's you know the standard kubernetes cluster we work in it because we don't have any special dependencies on it. Um in tanzu we haven't done any testing with that. We've tested on you know google cloud on azure on aws and then on you know. "36:08.27 greybeards And." "36:12.96 Tad Lebeck On on rancher and and and and um and open shift in the data center and as well as the Upstream version." "36:14.19 greybeards Open shift and all that stuff. Yeah" right right? right? right? So I saw somewhere in your your blog or or website that is of 1 line installation thing. How does this work. And those insillations take more than 1 line I'll have to say. "36:35.24 Tad Lebeck So It's it's It's the classic. You know there's a lot of magic hidden mind that 1 line in terms of you know" look. There's that's a script that actually takes and and you know. Pulls down from the repository. The different microservices you need and make sure they get installed and things like that. So. It's It's not you know 1 line per se it's a script that actually manages that for you. So The uners be 1 line but there's a lot more that goes on there. "36:59.42 greybeards Right? And the and the free service is that how is that limited or is it limited in time or limited in space or." "37:09.19 Tad Lebeck So It's It's limited to the number of node the number of data Nodes. So The idea is you know like a small cluster you can operate it for free for infinity. Um. But if you want to start scaling it up and having a lot more data nodes and and stuff like that then you should come back and talk to us now now the thing is right? You know for that whole model. You really want to show the value and if you hide it behind a paywall. They can't really undercover what the product can really do and we." "37:26.90 greybeards But there's no functionality gates or anything like that That's inhibited when you do this." "37:42.67 Tad Lebeck We viewed as you know" look they if they start using it. They're going to eventually grow to what they need to do more. "37:45.69 greybeards Right? right? right? right? right? right? right? Talk to me about operations of the system I mean is it. And is it all Api driven.. There's no gui at all or is the gui associated with it tells you you know how much storage you're consuming and whether you're going to go out run out of storage or anything like that." "37:57.44 Tad Lebeck Yeah. So So the the answer is it's all through a rest api and we have a ui that communicates to the product in that same rest Api. Um" and so that so we have prometheus and Grafana in the background and we can you know monitor on that. Um. "38:10.76 greybeards And using Graphana or something like that to." "38:21.70 Tad Lebeck And then we use you know the Ui is written I think in in angular. Um" and you know so you know you can you can look at and view in what you want to? do you can specify a teleportation you can say look I want to create a a backtrack time. You know time travel back on this volume. Through the Ui or you can do it through the rest apis. Okay. "38:41.44 greybeards Ah" okay okay and you're pretty well integrated in the cicd tools and stuff like that or. "38:48.17 Tad Lebeck Well so we're we're working on that right? So we What we do is we found that? Um" when we start talking people. There's real interest in this and so we've been working with you know. And we don't have a a Jenkins tool or or a circle Ci tool but we do have the ability to integrate with them and we're talking to some folks who are doing kind of devops in a box where they want to integrate us into their solution I can't talk about who that is just yet. But the idea is that we can. We can help you integrate it into your. "39:13.84 greybeards Right." "39:22.15 Tad Lebeck Pipeline. It's not a product because everybody's pipelines are their own pipelines so we help you integrate it into it or we'll work with partners who can actually do that for you." "39:27.92 greybeards Right? You mentioned partners you you guys do anything with like managed service partners. Our multi-tenancy I guess is the question That's typically asked in the scenario right? Do you guys support multi-tenancy." "39:33.78 Tad Lebeck It were early in the prospects. But there. Yeah" yeah yeah not as well. We'd like to yet. Yeah what's the part was Kubernetes doesn't really do multitenancy so you have to kind of like skin that cat each time. "39:43.74 greybeards And what would the even look like in Kubernetes I have no idea." "39:53.44 Tad Lebeck Um" and it depends on the managed service provider. So in the 1 case we have somebody where they're actually doing multi-tenancy but it was how they did it and we worked with them to make sure we worked with that because because Kubernetes didn't tackle multi-tenancy. There isn't a oh yeah we we just do kubernetes multi-tenancy you have to kind of. "40:01.89 greybeards All right." "40:13.16 Tad Lebeck Go look at how they solved it and you know what roles they defined and you know how they're doing that. Um" but yeah you have to you have to support it somehow. Um we're we expect to make a lot more progress than that as we go forward because we think that you know in this kubernetes universe. "40:16.28 greybeards Right? right." "40:33.80 Tad Lebeck The biggest problem people have is skill set shortage and so turning to trusted advisors like managed service providers and folks to help them implement and so it's It's the logical thing to work with them because they're the ones that customers are turning to for help. You know there's the really advanced shops who know what they're doing but by and large the the bulk of the market is really looking for help on on this." "40:36.76 greybeards Right." "40:43.96 greybeards M." "40:52.40 greybeards That brings up a question about professional services. Do you guys have a professional services organization that that people can call on to to help you know deploy install and optimize these sorts of solutions." "40:52.78 Tad Lebeck And escape." "41:00.87 Tad Lebeck He shows I'm not trying to be asked nine. We will help you do that We don't have an organization per yet but we will build that out and so we you know we handle it now and you know's I wouldn't say it's a professional service organization but we do you know." "41:11.12 greybeards I Got you." "41:20.64 Tad Lebeck We we see that as a need that you have to do to help people be successful and that's we're a startup so we're gonna help be successful." "41:24.10 greybeards And I I didn't see anywhere where you guys talked about open source. So I'm assuming iron air is all proprietary code that sort of thing. okay okay I think I'm about exhausted here. Tad anything you'd like to say to our listening ons before we close." "41:34.00 Tad Lebeck Yep" that's safe to say with those. "41:39.99 Tad Lebeck Know Yeah I mean I think that you you you already made the intro of like youll check out our website if you see you know" try it out. Um it's a really cool product and you know we'd love to have you kind of investigat it some more but um no I think great you you you did a good job of poking poking in all little spots on it. "41:56.77 greybeards Okay" ah good. Well this has been great. Ted thank you very much for being on our show today and that's it for now by tad until next time. "41:58.50 Tad Lebeck Ah" enjoyed it. "42:02.26 Tad Lebeck Thank you right? by ray."