From patchwork Mon Nov 27 20:49:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marlon Rodriguez Garcia X-Patchwork-Id: 35256 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AFB3C4167B for ; Mon, 27 Nov 2023 20:49:31 +0000 (UTC) Received: from mail.savoirfairelinux.com (mail.savoirfairelinux.com [208.88.110.44]) by mx.groups.io with SMTP id smtpd.web10.111711.1701118160516758817 for ; Mon, 27 Nov 2023 12:49:21 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@savoirfairelinux.com header.s=DFC430D2-D198-11EC-948E-34200CB392D2 header.b=oYaAi22U; spf=pass (domain: savoirfairelinux.com, ip: 208.88.110.44, mailfrom: marlon.rodriguez-garcia@savoirfairelinux.com) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 0DB329C2797; Mon, 27 Nov 2023 15:49:19 -0500 (EST) Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10032) with ESMTP id LWMAb9Sj95ay; Mon, 27 Nov 2023 15:49:17 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.savoirfairelinux.com (Postfix) with ESMTP id 598089C325A; Mon, 27 Nov 2023 15:49:17 -0500 (EST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.savoirfairelinux.com 598089C325A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=savoirfairelinux.com; s=DFC430D2-D198-11EC-948E-34200CB392D2; t=1701118157; bh=hUpStVXYsY61plPujuhH6hRXr2iCI7OV6eXxuIo0bwI=; h=From:To:Date:Message-Id:MIME-Version; b=oYaAi22UkVlkgGR7fOvWbuHefASpz4UwGJrVu6I8ShjSxHejjMQwiizEPcfi0oVBP CJsj8fNdKoz4KnJCoxmuwlX8ipiPk/na4BDX2oVUJo77TV8JRrNXQHnW549Uzw35zf 2Eg3K1YZWhF7dREOITIbXsNi0T/MwKrogQtol27l9HcZz5X+9UYucRMIxnt1kkHKNb lcBOM7hrD6xyZ5NMV2YruusyEoweVpTDeGEM5PRV7eRKA5mXniV8dViw5yAf7F3tiB IdJV5Ma49zHuJWxLHkpkFwleQqc/imH62ed5HZsLmZEWohO4A5B3vlSZGk26BW+uw3 B3jz6PzxojILA== X-Virus-Scanned: amavis at mail.savoirfairelinux.com Received: from mail.savoirfairelinux.com ([127.0.0.1]) by localhost (mail.savoirfairelinux.com [127.0.0.1]) (amavis, port 10026) with ESMTP id HT3UbzH-Wuvd; Mon, 27 Nov 2023 15:49:16 -0500 (EST) Received: from savoirfairelinux.ht.home (modemcable141.201-58-74.mc.videotron.ca [74.58.201.141]) by mail.savoirfairelinux.com (Postfix) with ESMTPSA id 6C3659C2797; Mon, 27 Nov 2023 15:49:16 -0500 (EST) From: Marlon Rodriguez Garcia To: bitbake-devel@lists.openembedded.org, toaster@lists.yoctoproject.org Cc: Marlon Rodriguez Garcia Subject: [PATCH] toaster: Added feat to import eventlogs files as builds in toaster Date: Mon, 27 Nov 2023 15:49:13 -0500 Message-Id: <20231127204913.94683-1-marlon.rodriguez-garcia@savoirfairelinux.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Mon, 27 Nov 2023 20:49:31 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/bitbake-devel/message/15577 New feature added to recreate command line builds that are made when toaster is off. The feature contains a new button on the base template to access a new template. A new model was added to register the information on the builds and generate access links A new form was added to include the option to load specific files This feature uses the value from the variable BB_DEFAULT_EVENTLOG to read the files created by bitbake Signed-off-by: Marlon Rodriguez Garcia --- lib/bb/ui/toasterui.py | 2 +- .../orm/migrations/0021_eventlogsimports.py | 22 ++ lib/toaster/orm/models.py | 9 + lib/toaster/toastergui/forms.py | 13 ++ lib/toaster/toastergui/templates/base.html | 3 +- .../templates/command_line_builds.html | 171 ++++++++++++++++ lib/toaster/toastergui/urls.py | 1 + lib/toaster/toastergui/views.py | 188 +++++++++++++++++- 8 files changed, 405 insertions(+), 4 deletions(-) create mode 100644 lib/toaster/orm/migrations/0021_eventlogsimports.py create mode 100644 lib/toaster/toastergui/forms.py create mode 100644 lib/toaster/toastergui/templates/command_line_builds.html diff --git a/lib/bb/ui/toasterui.py b/lib/bb/ui/toasterui.py index ec5bd4f1..6bd21f18 100644 --- a/lib/bb/ui/toasterui.py +++ b/lib/bb/ui/toasterui.py @@ -385,7 +385,7 @@ def main(server, eventHandler, params): main.shutdown = 1 logger.info("ToasterUI build done, brbe: %s", brbe) - continue + break if isinstance(event, (bb.command.CommandCompleted, bb.command.CommandFailed, diff --git a/lib/toaster/orm/migrations/0021_eventlogsimports.py b/lib/toaster/orm/migrations/0021_eventlogsimports.py new file mode 100644 index 00000000..328eb575 --- /dev/null +++ b/lib/toaster/orm/migrations/0021_eventlogsimports.py @@ -0,0 +1,22 @@ +# Generated by Django 4.2.5 on 2023-11-23 18:44 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('orm', '0020_models_bigautofield'), + ] + + operations = [ + migrations.CreateModel( + name='EventLogsImports', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('name', models.CharField(max_length=255)), + ('imported', models.BooleanField(default=False)), + ('build_id', models.IntegerField(blank=True, null=True)), + ], + ), + ] diff --git a/lib/toaster/orm/models.py b/lib/toaster/orm/models.py index 1098ad3f..19c96862 100644 --- a/lib/toaster/orm/models.py +++ b/lib/toaster/orm/models.py @@ -1868,6 +1868,15 @@ class Distro(models.Model): def __unicode__(self): return "Distro " + self.name + "(" + self.description + ")" +class EventLogsImports(models.Model): + name = models.CharField(max_length=255) + imported = models.BooleanField(default=False) + build_id = models.IntegerField(blank=True, null=True) + + def __str__(self): + return self.name + + django.db.models.signals.post_save.connect(invalidate_cache) django.db.models.signals.post_delete.connect(invalidate_cache) django.db.models.signals.m2m_changed.connect(invalidate_cache) diff --git a/lib/toaster/toastergui/forms.py b/lib/toaster/toastergui/forms.py new file mode 100644 index 00000000..a87e5391 --- /dev/null +++ b/lib/toaster/toastergui/forms.py @@ -0,0 +1,13 @@ +# +# BitBake Toaster Implementation +# +# Copyright (C) 2013 Intel Corporation +# +# SPDX-License-Identifier: GPL-2.0-only +# + +from django import forms +from django.core.validators import FileExtensionValidator + +class LoadFileForm(forms.Form): + eventlog_file = forms.FileField(widget=forms.FileInput(attrs={'accept': '.json'})) diff --git a/lib/toaster/toastergui/templates/base.html b/lib/toaster/toastergui/templates/base.html index 041448d1..e90be696 100644 --- a/lib/toaster/toastergui/templates/base.html +++ b/lib/toaster/toastergui/templates/base.html @@ -132,7 +132,8 @@ {% if project_enable %} New project {% endif %} - + Import command line builds + diff --git a/lib/toaster/toastergui/templates/command_line_builds.html b/lib/toaster/toastergui/templates/command_line_builds.html new file mode 100644 index 00000000..5b085e33 --- /dev/null +++ b/lib/toaster/toastergui/templates/command_line_builds.html @@ -0,0 +1,171 @@ +{% extends "base.html" %} +{% load projecttags %} +{% load humanize %} + +{% block title %} Import Builds from eventlogs - Toaster {% endblock %} + +{% block pagecontent %} + +
+
+
+
+
+
+

Import command line builds

+
+ {% if import_all %} +
+ +
+ {% endif %} +
+
+ {% if messages %} +
+ {% for message in messages %} +
{{message}}
+ {%endfor%} +
+ {% endif %} +
+

Import eventlog file

+
+ {% csrf_token %} +
+
+ +
{{ form.eventlog_file}}
+
+
+
+ +
+
+
+
+
+ +
+
+

Import eventlog files from directory + + + + + + +

+ {% if files %} +
+ + + + + + + + + {% for file in files %} + + + + + {% endfor%} + +
NameAction
+ {{file.name}} + + {% if file.imported == True and file.build_id is not None %} + Build Details + {%else %} + + + + {%endif%} +
+
+ {% else %} +
+
Sorry - no files found
+
+ {%endif%} +
+
+
+
+
+ + + +{% endblock %} diff --git a/lib/toaster/toastergui/urls.py b/lib/toaster/toastergui/urls.py index bc3b0c79..62629494 100644 --- a/lib/toaster/toastergui/urls.py +++ b/lib/toaster/toastergui/urls.py @@ -95,6 +95,7 @@ urlpatterns = [ # project URLs url(r'^newproject/$', views.newproject, name='newproject'), + url(r'^cmdline/$', views.CommandLineBuilds.as_view(), name='cmdlines'), url(r'^projects/$', tables.ProjectsTable.as_view(template_name="projects-toastertable.html"), name='all-projects'), diff --git a/lib/toaster/toastergui/views.py b/lib/toaster/toastergui/views.py index cc8517ba..a75fba47 100644 --- a/lib/toaster/toastergui/views.py +++ b/lib/toaster/toastergui/views.py @@ -6,24 +6,36 @@ # SPDX-License-Identifier: GPL-2.0-only # +import ast import re +import pickle +import codecs +import subprocess + +import bb.cooker +from bb.ui import toasterui from django.db.models import F, Q, Sum from django.db import IntegrityError -from django.shortcuts import render, redirect, get_object_or_404 +from django.shortcuts import render, redirect, get_object_or_404, HttpResponseRedirect from django.utils.http import urlencode from orm.models import Build, Target, Task, Layer, Layer_Version, Recipe from orm.models import LogMessage, Variable, Package_Dependency, Package from orm.models import Task_Dependency, Package_File from orm.models import Target_Installed_Package, Target_File from orm.models import TargetKernelFile, TargetSDKFile, Target_Image_File -from orm.models import BitbakeVersion, CustomImageRecipe +from orm.models import BitbakeVersion, CustomImageRecipe, EventLogsImports from django.urls import reverse, resolve +from django.contrib import messages + from django.core.exceptions import ObjectDoesNotExist +from django.core.files.storage import FileSystemStorage +from django.core.files.uploadedfile import InMemoryUploadedFile, TemporaryUploadedFile from django.core.paginator import Paginator, EmptyPage, PageNotAnInteger from django.http import HttpResponseNotFound, JsonResponse from django.utils import timezone +from django.views.generic import TemplateView from datetime import timedelta, datetime from toastergui.templatetags.projecttags import json as jsonfilter from decimal import Decimal @@ -32,6 +44,10 @@ import os from os.path import dirname import mimetypes +from toastergui.forms import LoadFileForm + +from collections import namedtuple + import logging from toastermain.logs import log_view_mixin @@ -41,6 +57,7 @@ logger = logging.getLogger("toaster") # Project creation and managed build enable project_enable = ('1' == os.environ.get('TOASTER_BUILDSERVER')) is_project_specific = ('1' == os.environ.get('TOASTER_PROJECTSPECIFIC')) +import_page = False class MimeTypeFinder(object): # setting this to False enables additional non-standard mimetypes @@ -1940,3 +1957,170 @@ if True: except (ObjectDoesNotExist, IOError): return toaster_render(request, "unavailable_artifact.html") +class EventPlayer: + """Emulate a connection to a bitbake server.""" + + def __init__(self, eventfile, variables): + self.eventfile = eventfile + self.variables = variables + self.eventmask = [] + + def waitEvent(self, _timeout): + """Read event from the file.""" + line = self.eventfile.readline().strip() + if not line: + return + try: + event_str = json.loads(line)['vars'].encode('utf-8') + event = pickle.loads(codecs.decode(event_str, 'base64')) + event_name = "%s.%s" % (event.__module__, event.__class__.__name__) + if event_name not in self.eventmask: + return + return event + except ValueError as err: + print("Failed loading ", line) + raise err + + def runCommand(self, command_line): + """Emulate running a command on the server.""" + name = command_line[0] + + if name == "getVariable": + var_name = command_line[1] + variable = self.variables.get(var_name) + if variable: + return variable['v'], None + return None, "Missing variable %s" % var_name + + elif name == "getAllKeysWithFlags": + dump = {} + flaglist = command_line[1] + for key, val in self.variables.items(): + try: + if not key.startswith("__"): + dump[key] = { + 'v': val['v'], + 'history' : val['history'], + } + for flag in flaglist: + dump[key][flag] = val[flag] + except Exception as err: + print(err) + return (dump, None) + + elif name == 'setEventMask': + self.eventmask = command_line[-1] + return True, None + + else: + raise Exception("Command %s not implemented" % command_line[0]) + + def getEventHandle(self): + """ + This method is called by toasterui. + The return value is passed to self.runCommand but not used there. + """ + pass + + +class CommandLineBuilds(TemplateView): + model = EventLogsImports + template_name = 'command_line_builds.html' + + def get_context_data(self, **kwargs): + context = super(CommandLineBuilds, self).get_context_data(**kwargs) + #get value from BB_DEFAULT_EVENTLOG defined in bitbake.conf + eventlog = subprocess.check_output(['bitbake-getvar', 'BB_DEFAULT_EVENTLOG', '--value']) + if eventlog: + logs_dir = os.path.dirname(eventlog.decode().strip('\n')) + files = os.listdir(logs_dir) + imported_files = EventLogsImports.objects.all() + files_list = [] + + # Filter files that end with ".json" + event_files = [file for file in files if file.endswith(".json")] + + #build dict for template using db data + for event_file in event_files: + if imported_files.filter(name=event_file): + files_list.append({ + 'name': event_file, + 'imported': True, + 'build_id': imported_files.filter(name=event_file)[0].build_id + }) + else: + files_list.append({ + 'name': event_file, + 'imported': False, + 'build_id': None + }) + context['import_all'] = True + + context['files'] = files_list + context['dir'] = logs_dir + else: + context['files'] = [] + context['dir'] = '' + + context['form'] = LoadFileForm() + context['project_enable'] = project_enable + return context + + def post(self, request, **kwargs): + logs_dir = request.POST.get('dir') + all_files = request.POST.get('all') + + imported_files = EventLogsImports.objects.all() + try: + if all_files == 'true': + files = ast.literal_eval(request.POST.get('file')) + for file in files: + if imported_files.filter(name=file.get('name')).exists(): + imported_files.filter(name=file.get('name'))[0].imported = True + else: + with open("{}/{}".format(logs_dir, file.get('name'))) as eventfile: + # load variables from the first line + variables = json.loads(eventfile.readline().strip())['allvariables'] + + params = namedtuple('ConfigParams', ['observe_only'])(True) + player = EventPlayer(eventfile, variables) + + toasterui.main(player, player, params) + event_log_import = EventLogsImports.objects.create(name=file.get('name'), imported=True) + event_log_import.build_id = Build.objects.last().id + event_log_import.save() + else: + if self.request.FILES.get('eventlog_file'): + file = self.request.FILES['eventlog_file'] + else: + file = request.POST.get('file') + + if imported_files.filter(name=file).exists(): + imported_files.filter(name=file)[0].imported = True + else: + if isinstance(file, InMemoryUploadedFile) or isinstance(file, TemporaryUploadedFile): + variables = json.loads(file.readline().strip())['allvariables'] + params = namedtuple('ConfigParams', ['observe_only'])(True) + player = EventPlayer(file, variables) + if not os.path.exists('{}/{}'.format(logs_dir, file.name)): + fs = FileSystemStorage(location=logs_dir) + fs.save(file.name, file) + toasterui.main(player, player, params) + else: + with open("{}/{}".format(logs_dir, file)) as eventfile: + # load variables from the first line + variables = json.loads(eventfile.readline().strip())['allvariables'] + params = namedtuple('ConfigParams', ['observe_only'])(True) + player = EventPlayer(eventfile, variables) + toasterui.main(player, player, params) + event_log_import = EventLogsImports.objects.create(name=file, imported=True) + event_log_import.build_id = Build.objects.last().id + event_log_import.save() + except json.decoder.JSONDecodeError: + messages.add_message( + self.request, + messages.SUCCESS, + "The file content is not in the correct format. Update file content or upload a different file." + ) + return HttpResponseRedirect("/toastergui/cmdline/") + return HttpResponseRedirect('/toastergui/builds/')