Home page / Businesses / Interview Armen Atayan is originally from Armenia. For a long time he was a business angel and invested in various startups. With the advent of GPT-3, he became interested in artificial intelligence, and then created the company Gen2B.ai, which helps implement AI in business processes. Six months ago, at the invitation of Bakht Niyazov, Armen moved to Kazakhstan to develop an open-source national language model - Irbis GPT. The official release of the product took place in June. At the moment, Irbis GPT has been downloaded more than 35 thousand times. Digital Business talked to Armen and found out what data the Kazakh language model was trained on and how much money was required to create it. We were also one of the first to test IrbisGPT. – How long have you been working in the field of artificial intelligence? – In 2022, when the GPT-3 model (davinci-002) appeared, I decided to send a request for access. At that time it was closed for public use, and it seemed that my application would be rejected. However, he received approval and began studying. By the way, our Head of AI was given access to GPT-3 back in July 2020. And OpenAI President Greg Brockman did it personally. In general, the topic of AI really inspired me. I understood
what changes await us in the future. I connected LLM with a chatbot on Telegram and showed my friends that it could answer questions competently. At that time, few people shared my delight. Afterwards, a separate AI model appeared for generating code, and I gained access to it. Then I realized that I wanted to develop professionally in this area. I stopped investing in other people’s startups and launched my own – Gen2B.ai. We are currently creating a platform that combines various AI products of our own design. For example, one of our products, Gen2Call, processes calls to the call center and also automatically identifies a user who left products in the cart on the site and did not make a purchase. – What motivated you to move to our country? – I lived in Bali for about three years, where I met Bakht Niyazov. One day he invited me to Kazakhstan at MOST Hub. I was imbued with the atmosphere of this place, saw how Astana Hub and the AIFC work. Also attended Digital Bridge 2023, where my team and I were able to freely communicate with ministers. In particular, I was able to present my solutions to Kanysh Tuleushin (First Vice-Minister of ICRIAP - Digital Business note), Sayasat Nurbek (Minister of Science and Higher Education - Digi note
tal Business) and Bagdat Musin (former head of the Ministry of Digital Development – note from Digital Business). I realized that Kazakhstan is exactly the country in which I will launch my life’s project. Moreover, in my opinion, Kazakhstan is the most promising state in the CIS, and it has the best chance of taking a leading position in the field of digital innovation and technology. – How did work on IrbisGPT begin? - From an idea. One day, Bakht and Pavel Koktyshev and I were discussing the future of the development of language models and their safety. I said that using models that are located on the servers of foreign organizations is unsafe and you need to create your own. Someone asked if we could make a Kazakh model? Gen2B had similar experience, methodology and expertise: we focused on creating LLMs for internal use by various businesses and governments as well. He answered yes, but asked for some time to confer with the team. However, just a few days later an announcement was made about the imminent appearance of IrbisGPT. Our Head of AI even panicked at first, because... understood how large a volume of tasks had to be completed in a short time. But we realized that we just need to take it and start working. Spent a lot of sleepless
nights and, despite the tight deadlines and lack of high-quality datasets, went through the full cycle of training language models. The results exceeded our expectations. Today IrbisGPT has a total of 35 thousand downloads. – What took the most time? – Mainly for research. We studied the experience of other countries, including the creators of Chinese-LLama (Chinese language model - Digital Business note). After test training, we realized that the option of taking a ready-made model, loading a data set there and doing fine-tuning (adaptation and additional training of a publicly available ready-made model for specific tasks - note Digital Business) does not suit us. The fact is that foreign neural networks were developed for the Latin group of languages, and therefore GPT-4 in Kazakh worked 5 times slower. The first thing we did was train our tokenizer to improve the efficiency of the model. Next, we downloaded a large number of texts in the Kazakh language so that the neural network based on them could find patterns faster and better. So, in IrbisGPT one Kazakh word is equal to one token. Now the same ratio is shown by GPT-4o in English, but there, for example, the word “rahmet” weighs three tokens. In more
In older versions, one token generally designated only one Kazakh letter. It took about three weeks to retrain, and our hypothesis that the model’s efficiency would increase after this was confirmed. – Where did you get the data in the Kazakh language? – A significant part of the dataset is information from the Kazakh Wikipedia, which is publicly available. We also took data from any open resources we found. I often had to use Google translator. A terrible dataset, in fact, since the accuracy of the translation leaves much to be desired. But our goal was to make a basic demo model that understands the grammar of the Kazakh language and correctly composes words into sentences while maintaining the general meaning. They couldn't hope for more in three weeks of training. At the same time, we requested the dataset of the National Corpus of the Kazakh language in order to fill IrbisGPT with a large amount of high-quality information. It is publicly available on the website qazcorpus.kz. We turned to the Ministry of Science and Higher Education of the Republic of Kazakhstan with a request to provide information in a convenient format. Unfortunately, we were refused. The reasons are still unknown. – Where do you plan to store the model? – Recently the Ministry of Digital Development
We announced the acquisition of a large supercomputer, and we were also promised access to it. We hope this will solve computing power issues in the future. – How many people are working on IrbisGPT? – The team is decentralized. There are people from Portugal, Serbia, Georgia, Thailand, Russia and, of course, Kazakhstan - about 15 people. We collaborate with the guys from Gen2B, MOST Hub and other specialists. After the presentation of the project at CEVF 2024, we received many applications from those wishing to join the development of IrbisGPT. We are now actively considering requests and will welcome new team members. – How much money is required to create a national language model? – If we want to create an LLM like ChatGPT, it requires significant financial investments, approximately $10 million or more. However, by focusing on creating smaller scale, more logical and applied models, the costs will be significantly lower. Millions of tenge have already been spent on the development of IrbisGPT. These are my personal funds, Bakhta Niyazov and MOST Holding. – Why does Kazakhstan need a national language model? Isn't it easier to wait until, for example, OpenAI implements this? – This is especially important for national security, so that
Citizens’ data did not leave the country and were processed internally. Thus, language models must operate in a closed loop. IrbisGPT was created just for this. IrbisGPT is also needed so that people who speak Kazakh have the opportunity to use their native language to obtain the necessary information. Usually, if there is a request, a person goes to Google and writes it in Russian or English, because this way they can get an answer faster. You can communicate with our model in Kazakh. In addition, it was important for us to prove that we can create large language models efficiently and quickly. This opens up routes to new markets. We are ready to work all over the world, not limited to the countries of Central Asia. - Speaking of national security. Should AI be regulated in Kazakhstan? - Undoubtedly. In my opinion, after realizing the possible consequences of the use of AI, regulation of this area around the world will be tightened. After all, there are already successful cases of how you can influence public consciousness with the help of neural networks. For example, the consulting company Cambridge Analytica used neural networks to influence the outcome of elections in various countries. In 2016, during Brexit, the company created
la various quizzes and personalized advertising encouraging people to vote for Brexit. Artificial intelligence helped target and distribute this content to specific psychographic groups. Thanks to materials like this, people voted for Brexit. This result was achieved using conventional neural networks, and now language models are appearing that understand human logic. This influence will definitely increase. Imagine, a large Kazakh company will introduce AI into the work of its call centers. Clients will call and make one or another choice during the conversation. In these dialogues, it will be possible to accurately track the decision-making algorithm of each person, and then develop tools to influence the entire nation. This, in our opinion, poses a rather serious threat. To prevent this, it is necessary to focus on creating your own, sovereign models, and also regulate the collection, distribution and sale of datasets. It is important to understand what data can be sold to other companies and countries, and for what purposes they will use it. - Let's move on to testing the system. What requests can IrbisGPT currently respond to? - This
hour IrbisGPT answers general questions: for example, he will tell you who the president of Kazakhstan is or explain the meaning of life. You can interact with the system on the website (for now only the desktop version is available) both by text and voice. We also trained a voice model to understand the Kazakh language and achieved an accuracy of 85%. However, the model can also work with context. For example, we recently downloaded the text of the Criminal Code of the Republic of Kazakhstan and asked several questions. The model retrieved the required information and responded correctly. Let's check how this works with a specific example. 1. In a special field, load the text that talks about biotechnologies. 2. We ask the question: “In what areas are biotechnologies used?”, and ask IrbisGPT. 3. After a few seconds we get the answer: “in medicine, agriculture and industry.” Now we have inserted a small text into the context field, but you can upload materials up to 8 thousand characters here. – What are your future plans for the development of IrbisGPT? – To release this version, it was necessary to create several iterations. Only after the fourth or fifth attempt did we get a satisfactory result. Therefore, now it is important to continue training the model by filling it with data. dl
This may require real translators into the Kazakh language in order to upload high-quality texts, and not crookedly translated ones using Google. Now IrbisGPT has 7 billion parameters, and, for example, the multimodal AI model Falcon 2, created in the UAE, has about 120 billion. Therefore, in order for IrbisGPT to be able to solve a wide range of problems, it is necessary to download approximately 30 billion tokens of high-quality data. We plan to create applied models for different purposes. For example, for emergency services, the educational sector, and government services. In the future, we would like to make a multimodal model that will be immediately trained on all types of data: audio, video, text and photos. This way we will achieve minimal delay. – The development of AI is proceeding very rapidly. Will neural networks be able to replace humans? – AI accurately automates routine work. Let's take pilots for example. Previously, they completely controlled the aircraft from start to finish, but now they are more like autopilot operators. In no way am I diminishing their contribution, but at the moment they have more control over the process and intervene if something goes wrong. I think the same thing will happen to programmers. They will become operators of neural networks, will form
Design the architecture and specify instructions for creating code, instead of writing it by hand. At the same time, professions that require human-to-human interaction, as well as where creativity and the process of creation are valued, will become more relevant. We admire Michelangelo's sculptures because we cannot imagine how one person could create such detailed works of art from a piece of stone. If we recreate such masterpieces with the help of robots and AI, it will not cause the same delight. Read also: From Kazakhstan to Germany: start from scratch and conquer Baden-Baden
