Programming LLMs for Business Application is Way Better Than ‘Tuning’ Them


In modern enterprises many employees rotinely have to perform tasks related to text understanding and basic manipulation. These include e.g. classification, routing, data extraction into spreadsheets/databases, and formatted summarization into a report. Typically, LLM based approaches to these tasks aim to 'train' or 'fine tune' the LLM to do the same, based on curated labeled data by the organisation itself, and/or external open source datasets from the relevant industry. However, the preparation of such datasets is an expensive, time consuming process. Another issue is that in cases the tuned LLM makes mistakes, it is not clear how many and which new labeled data is required to solve the issue. This makes actual commercial use difficult for many use cases.

Humans, of course, can accomplish the same tasks by being given a written or oral policy. In the NEC research labs, we have created an LLM based process of converting policies into 'prompt ensembles' that are used to effectively 'program' the existing, untuned or lightly tuned LLM to give discrete, constrained-by-prompt answers which are then aggregated and filtered to yield the final result. Constraining the results by breaking up the policy to a 'prompt ensemble' prevents the problem of hallucinations as the LLM answers are discrete or very short. At the same time, the accuracy level is increased by using partially overlapping/repeating prompts. When the resulting system does not implement the policy as desired, or the corporate policy itself is changed, it is easy to locate and modify/add the relevant prompts, and verify the mistake will not be repeated.

We will show sample cases where we have successfully employed this method to several common problems: e.g. fine text classification of emails on the Enron dataset, and compliance verification on contracts and budget related corporate texts. The process is not strongly coupled to the use of a specific LLM, nor to a specific language. As LLMs become stronger, our method directly benefits from that.


Tsvi Lev is the Managing Director of NEC Corporation's Israeli Research Center, engaged in AI applications for Digital Transformation, Cyber Defense, Safety and Healthcare.

Before that Tsvi was VP of Strategy at Amdocs, leading the Amdocs team in the AT&T Foundry, an Open Innovation Hub.

Previously, Tsvi managed the R&D Center of Samsung Electronics in Israel, devoted to innovation and cutting-edge development for Samsung’s future consumer electronics and cloud products.

As an Entrepreneur in Mobile Multimedia and Computer Vision, and sold his company to Emblaze Systems.

Tsvi holds a Masters in Theoretical Physics and an MBA, has over 80 patents to his name, and is a frequent speaker at events related to technological innovation.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google